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Preface 



For the twenty-sixth year, the Research and Theory Division of the Association for Educational Communications 
and Technology (AECT) is sponsoring the publication of these Proceedings. This is Volume #2 of the 2b'* Annual 
Proceedings of Selected Papers On the Practice of Educational Communications and Technology presented at the 
National AECT Convention in Anaheim, CA. Copies of both Volume #1 and Volume #2 were distributed to 
Convention attendees on compact disk. Volume #2 is also available on microfiche through the Educational 
Resources Clearinghouse (ERIC) System. 

This volume contains papers dealing primarily with instruction and training issues. Papers dealing with research and 
development are contained in the companion volume (26* Annual, volume #1) which also contains over 60 papers. 

REEEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from proposals 
before they were submitted to referees for review. Approximately fifty percent of the manuscripts submitted for 
consideration were selected for presentation at the convention and for publication in these Proceedings. The papers 
contained in this document represent some of the most current thinking in educational communications and 
technology. 

M. R. Simonson 
Editor 
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Solving on-the-job problems in modern work environments: Using Electronic 

Performance Support Systems 

Omur Akdemir 
Syracuse University 

Abstract 

Electronic Performance Support Systems (EPSS) have opened a new avenue for instructional designers. 
Organizations quickly understood their values and started to get benefit of using EPSS to solve on-the-job problems. 
This paper provides a brief overview of the EPSS and illustrates the advantages of using EPSS in modem work 
environments. 



Introduction 

The world is changing faster than ever before. Especially developments in computer technology accelerate 
this change. Corporations are faced to integrate new technologies faster to survive in today’s corporate environment. 
This requires their employees to learn the operation of more complex products. As a result of this process of change, 
employees must maintain current levels of operational effectiveness while learning and retaining new complex 
processes and skills. In the context of changing environments, many employees are becoming multifunctional. 
Employers expect their employees to do many more tasks at irregular intervals (Spaulding & Dwyer, 1999). Thus, 
training becomes inevitable for organizations whose employees use new products. Many organizations devote their 
valuable time to train their employees. There is no doubt that spending time for training improves productivity if the 
training is designed and implemented properly. However, organizations loose money and workforce when their 
employees attend training sessions. 

Once upon a time, employees learned exclusively within actual job contexts while doing real work. In other 
words, they started performing simple tasks and progressed along the proficiency curve with personal and 
individualized guidance by an expert (Gery, 1991). Today’s business world makes it impossible to organize and 
offer such a training approach because of the number of employees that need to be trained at a time. There is no way 
to find this number of experts for each trained employee. Moreover, every new introduced product has many 
features and even after the training many of them, if they are not practiced regularly, have a risk to be forgotten. 

Electronic Performance Support Systems (EPSS) have opened a new avenue for instructional designers. 
Organizations quickly understood their values and started to get benefit of using EPSS in their 
organizations to solve on-the-job problems. The foundations of EPSS are in fields such as cognitive 
psychology, instructional design, and interface design. The late 1980s witnessed emerge of EPSS in 
business to enhance performance in complex and changing environments and to respond to concerns 
regarding the cost effectiveness of training (Chiero, 1996). This paper provides a brief overview of the 
EPSS and illustrates the advantages of using EPSS in modem work environments. 

What is EPSS? 

The EPSS have different names including Integrated Performance Support, Automated Performance 
Support, Performance Support Systems, and Performance Support Tools. Even though the labels are different, the 
purpose is the same; to improve performance in the work environment. According to Miller (1996) “[a]n electronic 
performance support system is any computer software program or component that improves employee performance 
by either: 

Reducing the complexity or number of steps required to perform a task (process simplification), 

• Providing the performance information an employee needs to perform a task, or 

• Providing a decision support system that enables an employee to identify the action that 
is appropriate for a particular set of conditions” (p.2). 




Where does EPSS fit? 

Before attempting to explain EPSS, it is beneficial to first look at where EPSS fit under the umbrella of 
technology-based learning solutions. The IBM e-leaming model helps to find where EPSS fits within the 
technology-based training solutions. 



Learning Lab 
Layer 



Collaborative Layer 



Interactive Layer 



Quickview Layer 



Mastery 

Put it all together, 

Oassroom, Mentoring, EPSS 

Applied skill 

Discuss it, practice it, with others 
Collaborative learning, 

Team rooms, on-line Mentoring 

Application 

Try it, practice it 
CBT, WBT, Simulations 

Knowledge & Awareness 

Read it, see it, hear it 
Web lectures. Info web sites 



Eigure 1 . The IBM e-learning model suggested by Mac Eke (2000) 



EPSS takes place at the Mastery level, which encompasses all other three levels, knowledge & awareness, 
application, and applied skill. Learners perform all the requirements of these three sub-levels. They read, see, hear, 
try, discuss, and practice. EPSS can provide activities, practices, and simulations, each of which reflects the context 
in which the learner invokes the component. This component is usually a type of computer-based or intelligent 
computer-assisted instruction with small modules that provide instruction related to a specific piece of the 
performance (Chiero, 1996). 



Why EPSS? 

According to MacEke (2000) “the focus is less on getting people in traditional classes than it is about. . . 

• Increasingly embedded learning in business processes and initiatives 

• Increasing overall productivity and competitiveness 

• Leveraging learning solutions as a way to solve problems 

• Eacilitating the transfer of knowledge” 

Based on this trend, it could be said that organizations are looking for a way to improve productivity when their 
employees are still working rather than attending traditional face-to-face training. In other words, organizations want 
their employees to perform certain tasks when they are required. It is obvious that if these tasks are performed in 
certain periods of time, learning inevitably occurs. By doing this, organizations are able to keep their employees in 
their work environment and help them to solve on-the-job problems when employees encounter them. Since the 
primary purpose of electronic performance support systems is to improve productivity by providing help to perform 
specific tasks, EPSS has become the popular way to solve on-the-job problems. 

Advantages of EPSS 

The main purpose of the all-training activities is to have people perform certain tasks. Training designers 
expend effort to reach this goal. However today’s work environments require too many tasks that need to be 
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performed. Training at one point becomes not the solution. Even well designed training sometimes does not reach its 
aim. There are different reasons why training does not become the solution and as a result EPSS are preferred. 

The complexity of the task is one of the reasons. Organizations frequently are increasing their expectations 
of their employees. One employee has to perform too many tasks, so employee has to learn these tasks first. 
However, because of the complexity of the tasks, it is difficult to memorize the necessary steps and perform them in 
a correct order. It is often desirable and possible to improve performance without necessarily promoting learning, to 
create expertise without necessarily creating an expert (Rosenberg, 1995). EPSS allows employees to access 
necessary information when it is required in the work environment, so employees do not need to worry about the 
complexity of the tasks. 

When EPSS are used to minimize rote learning, information overload for workers is reduced. Using EPSS 
also allow workers to concentrate on higher cognitive processes (Gilbert, 1998). This increases the self-confidence 
of employees and as a result the productivity of the work done. 

The tasks are constantly being changed. Thus, employees have to adapt themselves these changing tasks. 
Training programs are organized to facilitate the adaptation process of employees to the recently introduced tasks. 
Since employees often do not perform tasks frequently, the knowledge and skill they learn in training have a risk to 
be easily forgotten. To really acquire and master a skill, it is not sufficient to simply listen to someone explaining it 
to you (MacEke, 2000). Thus, it is beneficial to master a skill when performing that skill. EPSS lets employees 
master certain skills while they are performing them in real work environments instead of attending training. 

Perhaps one of the most important advantages of EPSS for organizations is that it reduces training expenses 
and the time spent for training employees. EPSS was developed by people look-ing for ways to impact performance 
directly, without the intermediate steps involved in instruction (Rosenberg, 1995). The technical advantages of 
delivering learning and performance support over an intranet include ease of distribution, ease of access, and use of 
existing infrastructure. These advantages offer immediate cost savings, as well as the promise of transforming work 
and learning as a result of increased connectivity (Gilbert, 1998). Consequently, organizations are able to save 
money because they do not need to organize training and so their employees do not need to attend training. 



Conclusion 

The complexity of the business world makes competition between companies more challenging. 

Companies have to adapt themselves to the changing world more rapidly than ever before. As a result of this 
changing world, the work environments have been changed rapidly. The faster the employees adapt these changing 
work environments, the more success companies will achieve. Traditional training methods sometimes do not 
alleviate the workload of training designers who help employees to adapt themselves to the changing work 
environment and tasks. EPSS have opened a new avenue for the designers to support on-the-job performance. Using 
EPSS offers several benefits. Employees can perform complex tasks without necessarily learning them. EPSS 
minimize rote learning and employees can consecrate on higher-level cognitive processes. Einally EPSS impact 
performance without the intermediate steps involved in instruction. 
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Time to Learn: The Use of Institutes in the Development of Faculty 

Technology Skills 

Cheryl A. Anderson 
University of the Incarnate Word 

Institutions large and small are struggling to find successful ways to encourage faculty to build technology 
into the curriculum. Kenneth C. Green, Director of the Campus Computing Project, reports that institutions consider 
the challenge of supporting faculty in the integration of technology in teaching to be the highest priority. He states, 
“Roughly one-fourth (24.3 percent) of the 2002 survey respondents cite instructional integration as the key IT issue 
for their institutions in the coming years (compared to 31.5 percent in 2001, 40.5 percent in 2000, and 29.6 in 1997)” 
(Green, 2002, p 25). Administrators of information technology rank “faculty development, support and training” as 
being among the top ten institutional strategic issues facing institutions of higher education. (Kobulnicky, Rudy, et. 
al, 2002). According to these researchers, understanding faculty culture is essential in the creation of successful 
faculty development programs. 

Unfortunately, faculty members face many barriers to the use of instructional technology. Butler and 
Sellbom (2002) found that the unreliability of technology; the time it takes to learn to use technology; questions 
about instructional appropriateness; and lack of support from the institution cause faculty to turn away from its use. 
In support of these findings, the presenter (Anderson, 2002) has found that faculty rank time to plan for technology 
to be a major hurdle for not using it at University of the Incarnate Word (UIW). In 2001, 53 percent of respondents 
to a survey on technology needs identified time as the primary reason for not using technology. More than one-third 
(37%) of the respondents cited limited access to equipment; and less than 10 percent identified lack of information 
about technology and being uncomfortable with technology barriers to technology use. 

Although UIW Office of Instructional Technology (OIT) offers workshops during the school year, the 
number of faculty participating in these sessions has not been excessive. This is due to a four-course workload 
requirement for faculty and their obligation to participate in committee work. It makes it difficult for faculty to find 
the time to participate in training. In response to faculty reported needs, the OIT staff implemented a technology- 
training program, which targets the barrier of time. The training strategy is to provide intensive, hands-on 
Technology Institutes during times of the year when faculty members have either a light teaching load or none at all. 
The first two-week institute was started in the summer of 1998 with a limited budget of $10,000. With the financial 
assistance of a Title V Hispanic Serving Institution grant awarded in 1999, the program has been expanded to 
include a Winter and Summer Institute. Two levels of training have been developed. With the implementation of 
the course management system Blackboard, a seminar in online pedagogy has been added. 

The goals of the training program are to have each faculty member create an instructional product using 
various media; create a collaborative learning environment among the faculty and the instructors; develop in faculty 
an awareness of UIW technology resources including equipment and human resources; have faculty learn to play, 
create and problem solve using technology; and to develop within each participant a level of comfort and self- 
sufficiency with the technology. Since 1998, over 80% of the fulltime faculty members (total 135) have received 
training through this intensive, hand-on experience. As a result, the University has reached a critical mass of trained 
faculty, who are putting their technology skills to use in the classroom. 

This paper will discuss the Technology Institutes in terms of the application process, format, content and 
evaluation. At the end, the author will make recommendations for other institutions looking for successful training 
strategies based on six years of experience. The results of several follow-up surveys which have been conducted 
with faculty one-year after their institute experience to determine what progress they have made with regard to 
integrating technology in their classroom teaching will be presented. 

Technology Institute participants are selected in a competitive process. In a two-page application, applicants must 
describe their interest in learning new technology skills and the anticipated impact of the skill development on 
classroom teaching and their particular academic program. They must identify an instructional problem that can be 
addressed using technology. Finally, their Deans must provide a recommendation. Only 10 faculty are selected for 
the Basic Training session and 8 are selected for the Advanced Training. The juried selection process makes 
attending the institute "special" for participants; and as a result, many faculty highlight it in their annual 
performance reviews. 
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Training in the summer is two weeks and the winter session is compressed into six-day period before the 
second term. A combination of lectures, demonstrations and hands-on practice is used to cover the curriculum. 
Although they have taught for years, many faculty have little formal training in instructional design so this is 
covered in the first day of class. Faculty are asked to identify a problem and an audience, write instructional 
objectives, create an outline and select a media before they start their project. Storyboard techniques are discussed 
and participants are required prepare a storyboard of their project by the end of the first week. The curriculum also 
introduces graphic design and web page design concepts. In addition, the trainers have created practice exercises for 
each piece of software used and faculty are given reference books as reinforcements. 

The last half of the training is dedicated to producing media projects. During the summer faculty may 
choose to do a Microsoft PowerPoint^M presentation or do a web site using Microsoft FrontPage^M^ depending on 
their own interests and comfort level with the technology. In the winter training, they create a project with both 
packages. The participants are required to discuss their projects with their colleagues using their storyboards. Then, 
each participant is expected to spend time during and outside of class developing a final product. The trainers work 
one-on-one with the faculty in helping to solve technical problems. In addition, the faculty help each other as they 
work through difficulties. 

On the final day of the institute, each participant is given time to display his or her project for their 
colleagues. When the showcase is complete, the participants vote for the works that demonstrate the "best of" in a 
number of categories. The institute closes with a luncheon and the awarding of certificates. 

Due to faculty requests, in the summer of 2001, the OIT opened an Advanced Technology Institute. It uses 
a similar two-week format. The application process is the same as with the basic training, which is a pre-requisite for 
the advanced training. The curriculum is different and more challenging. Participants learn to use Toolbook 
Instructor II, Sound Forge, Photoshop and a digital video editing system called Dazzle. Instead of working on 
individual projects, the participants work in two groups of four people. Each group puts together a multimedia 
tutorial that has already been scripted and storyboarded. The graphics and video clips have been prepared ahead of 
time. The task is to pull it all together into an 80-slide tutorial. Participants must learn to use Toolbook’ s Action 
Editor to make the tutorial interactive. They record music and narration and edit the video into a small movie that is 
into the tutorial. The team approach allows the group to finish the project within two weeks. Eaculty can focus on 
learning the multimedia tools rather than worrying about the content. By sharing the workload, participants with 
weaker skills can lean on faculty with stronger technology skills to get the work accomplished. 

Eaculty rewards have been an important part of the training program. Eor the first two years, the institute 
could only afford to give participants small tangible rewards like books, T-shirts, certificates and luncheons. The 
biggest reward for the first two years was a digital camera for each faculty member. The limited TI budget ($580 per 
person) did not make it possible to compensate faculty for their time. Eaculty were allowed to spend $150 on a piece 
of hardware or software of their own choosing. In evaluations, faculty were overwhelmingly positive towards these 
small benefits. The University received a Title V Grant for Hispanic Serving Institutions in 1999 that provided, in 
addition to the standard rewards, a $1,000 stipend for each faculty member. We also purchase $500 worth of 
hardware or software for each participant. 

The addition of the Title V money certainly has peaked faculty interest in the TI, but so has the quality of 
the experience. The OIT has conducted evaluations each year that show the faculty's overall reaction to the institute 
as being very positive. All of the participants have stated that they would "highly recommend" the experience to a 
friend. In a 2003 Technology Needs Assessment of fulltime faculty, 52% indicated that having an opportunity to 
participate in the Technology Institutes was most beneficial to their learning about technology. 

Eor the past three years, the Title V office has done a follow up survey with former Technology Institute 
participants approximately one year out from their institute experience. The purpose is to find out how their 
participation in the institute has affected their teaching practices with technology. In the most recent follow up, 25 
participants in the 2002 Institutes were surveyed using multiple answer and open-ended question instrument. The 
response rate was 64%. Eorty-four percent of the respondents indicated that they were using technology more than 
before the institute and 50% responded that they were using it somewhat more and no one indicated their use was 
less. In addition, 81% indicated they were using PowerPoint and 75% were using Blackboard in their courses. 
Eaculty indicated that they were assigning students to use the Internet (81%) and other software (50%) as part of 
class. In addition, 69% indicated that they had taught students to use technology needed for class. When asked 
about what their reasons were for NOT using technology this year, only 12% of the respondents indicated that they 
had a concern about the compatibility of using technology with their course content. The number one reason was 
that they were too busy with other work (50%). However, faculty remain interested in learning more about 
technology in that 62% indicated that they had attended a campus technology workshop, 44% had given a 
conference presentation using technology and 31% reported that they were using technology to conduct research. 
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When asked if the TI experience had changed their practices with technology for the better responded, one faculty 
member stated, “I have seen a definite change in the way I use technology. I am not skeptical about investigating 
and showing others how to use technology even if I make an occasional mistake. I am not afraid of technology 
anymore.” Another stated, “My level of comfort in using technology in the classroom has increased considerably; 
and I believe this has been transferred to my advisees and other students.” These are all indications that faculty who 
attended the training come away feeling more comfortable with the technology and are integrating what they have 
learned into their teaching practices. 

The evaluation results have reinforced our belief that UIW has developed a winning strategy for technology 
skill development. As of the summer of 2003, 1 10-fulltime faculty have been through Technology Institutes. This 
training has changed how faculty use technology in teaching. 

It does take hard work to put together a program such as UIW's Technology Institute. It can be done even on a 
limited budget. Our recommendations for other institutions of higher education thinking about faculty training are 
these. 

1 . Plan an institute during the time periods where the workload is lighter for faculty or during times in 
between semesters. 

2. Formalize the application process; make it competitive and peer reviewed for added value; 

3. Keep the classes small, more than 10 participants cuts down on individualized attention; 

4. Model the behavior by using multimedia to teach faculty the skills; 

5. Change the technology with the times and teach the most current version of the programs; 

6. Give the participants copies of the software for their office so that once they learn to use it, they have ready 
access to it; 

7. Do not under estimate the little rewards such as T-shirts, lunches and books; 

8. Give faculty something that they would never buy themselves such as a digital camera; 

9. Hire trainers who are patient, helpful and sensitive to the vulnerability of a faculty member who has a 
doctorate but who cannot right click a mouse; 

10. Have people on staff who can support the faculty after the workshop, and 

1 1 . Evaluate the workshops and conduct follow up evaluations so improvements can be made to the program. 

Finally, the OIT staff has come to realize that the greatest benefit for the participants of this institute is that it 
energizes faculty. It makes them feel more positive about their work and it gives them an appreciation of what it is 
like to be a beginner — a learner. Most faculty have been experts in their field for years. Such an experience can 
make them more sensitive to their own students' needs. As one faculty put it so eloquently in her 1998 evaluation, 
"This program expanded my knowledge of the 'mystery' of technology as well as the vast potential available for 
teaching! However, this experience also made me realize once again some of the challenges my students face in 
every class. In other words, learning became real once again." 
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Evaluation is a critical component to the success of all instructional systems. Traditionally, the evaluation 
process is used to identify strengths and weaknesses in instructional content (e.g., activities) by assessing outcomes 
such as the alignment of the content and instructional goals and objectives with one another. In organizations, 
evaluation of Web-based training ensures that goals and objectives are met, that learners acquire knowledge and 
skills from the content and that they can apply what they have learned to their job. In Web-based learning, it is also 
important to consider other factors that affect instructional content including the presentation of content (i.e., 
interface design and information architecture), clustering/grouping of content, and accessibility (including for people 
with disabilities) of information on the screen. 

The fields of education, psychology and technology each offer models for evaluating Web-based learning, 
and each have a different focus. By combining the strategies from each discipline, the definition of evaluation is 
broadened to include user assessments (e.g., learner validation of content and system features, instructor review of 
content and system features) and subject matter reviews (e.g., content reviews by subject matter experts, system 
reviews by system engineers, interface assessments by interface experts). In this way, the evaluation process can be 
maximized to provide a more thorough assessment of the quality of Web-based training in organizations from a 
learner, subject matter expert, and system perspective. Organizations can benefit from considering these multiple 
perspectives since this holistic approach is more likely to uncover what is effective and not effective in the training 
so that resources for revising training can be effectively targeted, and so that effective training programs continue to 
be developed. 



Perspectives on Evaluating Web-Based Learning 

To describe and eventually blend the three approaches, it is helpful to work from a common model in order 
to see differences among the perspectives. The educational approach will be used to anchor the discussion of the 
different approaches because it is the Analysis, Design, Development, Implementation and Evaluation (ADDIE) 
model, or variants of it that is commonly used to create instruction. 

Educational Perspective on Evaluating Web-Based Learning 

Evaluation from an educational perspective focuses on alignment of goals and objectives with what training 
is needed, and assessment of the content and activities to ensure they support the goals and objectives. The field of 
education presents a structure suggesting that evaluation be conducted during training design and development to 
ensure constant alignment of the content with goals and objectives, and after implementation to assess student 
learning and effectiveness of training materials. Specifically, most versions of the ADDIE model suggest a process 
during instructional creation so that after gathering needs for the training (from learners, supervisors and the 
organization) in the analysis phase, training content is designed (e.g., determining the cognitive level of learning, 
modules for the content) and developed (e.g., creation of content) and evaluated iteratively as “chunks” of training 
content are created. Once the training is complete and implemented, evaluation occurs again. While not all 
organizations do so, evaluating training throughout the instructional design process is important for assessing the 
congruence among content, goals, objectives and determining whether student learning occurred. 

Locus of Evaluation 

As demonstrated in the exhibit below, evaluation from an educational perspective occurs at several places 
within the ADDIE model’s approach. As organizations design, develop and implement Web-based training, they 
may include a number of evaluation measures related to assessing return-on-investment, which is directly related to 
the quality of the training content. Evaluation during design and development - often called formative evaluation 
since it is conducted as the Web-based training is being “formed” or created - focuses on ensuring that content 
meets the goals of the course and the needs of the learners through learner validation and review of materials and 
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activities by subject matter experts. Subject matter experts are guided during their evaluation by their understanding 
and experience with the content. Learners, on the other hand, are guided by the extent to which the content and 
activities facilitate their learning. 




Figure 1. Educational Perspective: Analysis-Design-Development-Impleinentation-EvaluationModel 
with a focus on evaluation. 



Training designers will create modules or lessons and have learners and subject matter experts review them 
prior to developing all of the training. This iterative process of design, development and evaluation not only reduces 
development costs, but also ensures that training content is constantly being adjusted as appropriate so that no major 
modifications will be needed after the entire training is developed. Training designers employ subject matter experts 
in the review of training goals and objectives, and assess the alignment of content and learner activities (e.g., 
facilitated chats, small group activities) planned with these goals and objectives. Subject matter experts also review 
student assessments to assess if they are measuring learning at the appropriate cognitive level (e.g., from knowledge- 
level or recall to problem-solving). Select learners within the target audience may be asked to review some 
materials and activities to ensure that the presentation of, and interaction with the content are interesting and 
relevant to them. The goal of formative evaluation from an educational perspective is to ensure accuracy and 
relevancy of the content, the inclusion of appropriate learner activities and assessments, and provide direction on 
changes needed to ensure training goals are met during training. 

When educational evaluation occurs after implementation of the Web-based training, it is called summative 
evaluation. Generally, summative evaluation is conducted by outside evaluators who were not involved in the 
creation of the training. The focus is on assessing the training against the outcomes associated with the training. 

Using evidence of student’s achievement (e.g., portfolios, assessments, papers, quality and quantity of 
online discussions), evaluators assess whether the training goals and objectives were achieved, and make 
recommendations for improving the training. 

In traditional educational environments, the ADDIE model works well and is fairly thorough. However, in 
web-based training environments, there are other factors that affect learning that should also be considered during 
evaluation such as the organizational impact of individuals who apply their training as examined from the 
perspective of psychology, and the human factors and system perspective of the technological perspective. 

Psychological Perspective on Evaluation 

The field of psychology also uses a structure for evaluating training outcomes. Kirkpatrick’s four levels of 
training evaluation criteria exemplify the psychological perspective on evaluation which are: reaction, learning, 
performance, and results. The first level, reaction, refers to what a learner thought of their training experience. 
Kirkpatrick’s second level, learning, refers to the measurement of knowledge, skills, and attitudes that were 
specified as training objectives. The third level, performance, refers to behavioral changes as a result of training. 
Finally, results, refers to the impact of the training on organizational objectives. 

Evaluating these levels is important for measuring desirability of the learning environment, learning, 
performance on the job and organizational change as a result of training. For organizations, documenting these 
types of outcomes can provide information such as how best to improve a training course, evidence that training is 
working well and should continue, or evidence that the cost of training is outweighed by the impact of the results. 

Focus of Evaluation 

This type of evaluation is typically conducted immediately after training delivery (reactions and learning levels) or 
at some point thereafter (performance and results levels). This is shown as summative evaluation in figure 1. 
Evaluations from the psychological perspective are typically conducted by evaluators that did not develop the 
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training. However, evaluations at the reaction and learning levels can be conducted by training developers or 
instructors. For each of the four types of outcomes there are specific questions that can be asked or data that can be 
gathered. 

The focus during reaction measurement is to determine if there are aspects of the training that can be 
changed to better suit the learner, from the learner’s perspective or the instructor’s perspective. Reaction measures 
typically ask learners questions about their overall satisfaction, perceived usefulness of the content, satisfaction with 
the course process and reactions to different aspects of technology. This information is then used to make 
improvements to the course content and process that will improve satisfaction, learning, or retention of knowledge 
in future iterations of the course. 

The purpose of measuring learning is to determine if the course objectives have been met and if the training 
is working as it was intended to work. Measures of learning typically assess the acquisition of knowledge and skills, 
or changes in attitude as a result of the training. These measures are typically administered upon completion of 
training, and can occur in a variety of forms such as a multiple choice knowledge test, hands-on testing, or measures 
of behavioral intentions. Information gathered about the learning that has taken place can be used to adjust course 
content to improve learning in future courses and to better facilitate transfer of training. 

Performance as a result of training in organizations can most directly be measured by job performance. Job 
performance can be measured by supervisory assessments of performance related to training objectives or by more 
objective measures of performance such as changes in productivity. Evaluating the extent to which training has 
changed job performance is a means of determining if training transfer has taken place, and a direct, measurable 
indicator of the impact of training. 

Results are usually organizational level outcomes. Results can be measured by such things as cost, 
turnover, absenteeism, accidents and morale as they relate to the goals of the training. Measuring the impact of 
training on factors such as these can lead to a direct translation of return on investment, as long as it can be 
determined that the impact on one of these factors is directly the result of training. 

Evaluating each level is important for measuring desirability of the learning environment, the amount of 
learning that has occurred, the impact of the training on job performance and organizational change as a result of 
training. Evaluating training outcomes can enable an organization to document what is working well and what 
needs to be changed to better meet the goals and objectives of training. Evaluation from the psychological 
perspective addresses some important aspects of organizational training and can be tailored to address Web-based 
training. However, it does not capture many of the technological aspects of training that are addressed by the 
technological perspective of evaluation, nor does it implement an iterative evaluation during development such as 
those employed by both educational and technological evaluation. 

Technological Perspective on Evaluation Evaluation from a technical perspective broadens the general definition of 
evaluation to not only include evaluation from a user (e.g., learner, instructor) perspective, but also from a system 
perspective (meaning testing the system to ensure functionality that supports the Web-based training) and interface 
perspective (to ensure that the way information is presented is usable). The software development life cycle is the 
generic model for approaching the creation and evaluation of Web-based training and other systems. It contains six 
basic steps conducted by instructional technologists and software design specialists (SDS) to build and assess the 
system including analysis (identifying organizational, instructor and learner needs for the software), design (layout 
of the pages and the architecture of the back-end systems that support it), code generation, testing, release (or 
implementation of the Web-based system) and support. These phases support the gathering of requirements for the 
Web-based training system, including database(s), interface(s), and design needed to support the online training. 

Similar to the educational perspective, evaluation of the Web-based system throughout the creation of the 
training is important for assessing the congruence between the requirements gathered and the system that is built to 
support learning. In this way, the training can be built to better match the training goals, instructors and learners 
needs. 

Focus of Evaluation Evaluation can only occur after a thorough gathering of software requirements in the analysis 
phase. The data gathered in analysis provides the measures against which all technological evaluation activities will 
occur. Next, organizations design Web-based training systems and generate the code that will support the learning 
environment, including test plan development, in order to assess the quality of the system as it relates to the 
requirements gathered. Eigure 2 exemplifies evaluation throughout the SDLC process. 
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Architecture, Wireframe and Mock-up Eualuations, Formal Technical Reuieuvs, Heuristic Reviews 

Figure 2 . Technological Perspective. Software Development Life Cycle with a focus on evaluation. 

Evaluation of the training by system design experts during the design phase focuses on ensuring that the 
system design and system architecture - or diagrams displaying the relationships among the database(s), 
interface(s), component(s), and other system(s) - are accurate according to the system requirements analysis phase. 
During this formal technical review, system design experts are guided in their evaluation by their understanding of 
the current systems’ needs and their experience with similar systems. 

Also in the design phase, interface designers begin laying-out the Web pages as wireframes. Wireframes 
are simplified outlines of pages describing the general location of the navigation, content and other features so the 
training content is presented consistently. Next, graphics designers begin creating images that are displayed on 
pages to create the look-and-feel of the training. These images will then be combined with the wireframes to create 
mock-ups, which are simplified HTML pages. During this process, a heuristic review takes place where wireframes 
and/or mock-ups are evaluated by outside interface design experts (who were not involved in the page designs) and 
by representative members of the target audience, including instructors and learners. Interface design experts are 
guided in their evaluation by the professional norms of interface design called heuristics, and instructors and learners 
are guided by their needs as they relate to the training. 

These evaluation efforts reduce development costs as the system is validated by users (including instructors 
and learners) before investing money and time for programmers to generate code. The earlier that design flaws are 
identified, the easier they can be remedied, and the less the training system costs to build. 

As the organization completes the design phase, the coding phase begins. Coding is the process of implementing the 
design including the interfaces, databases, and incorporation of other systems and components needed by the Web- 
based training. Systems designers, programmers and interface architects work together to create the training system, 
however the majority of the work is conducted by programmers as they follow the design (specified in the 
wireframes and mock-ups) and write code. During this phase, code reviews are performed by outside evaluators 
(i.e., systems designers and programmers) who were not involved in creating this particular training system. As 
components or “chunks” of the system are written, the code is reviewed for thoroughness and accuracy, but the code 
is not yet exercised. 

This evaluation process is similar to the educational perspective wherein content is developed in modules 
and lessons, which are then reviewed by subject matter experts and learners before the entire training content is 
developed. Systems designers and programmers are guided in their evaluation of the code by the design established 
in the requirements documents, professional norms for writing code in various languages, and their experience with 
similar systems. Recommendations as a result of the code review are incorporated as coding continues. This 
iterative process continues until the coding for the Web-based training system is complete. 

Finally, testing is a phase where the system components are evaluated, and then the components are 
assembled so that the entire system is evaluated. Three main types of testing occur in this phase; 1) unit testing, 2) 
integration testing and 3) validation/acceptance testing. Unit testing is conducted to determine the integrity of 
related chunks of code by exercising code at the smallest logical level or module. Programmers who were not 
involved in the creation of the system evaluate the units for logic errors, inefficiencies in code, and completeness. 

Next, the system undergoes integration testing to evaluate functionality of the system between and among 
units by combining units together - beginning with two units and continuing until all units are combined - until the 
system is fully assembled. Again, like unit testing, outside evaluators (in this case programmers) test during 
integration by looking at a variety of factors (e.g., communication among units, synchronization among units, and 
proper interface implementation). 
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Finally, the training system is evaluated during validation/acceptance testing by the client, stakeholders, 
and users (e.g., instructors, learners) to ensure the entire system is performing according to the goals of the system. 
This evaluation is guided by the client, stakeholders’ and learners’ expectations for the system as indicated during 
the analysis phase. During the testing phase, any problems found may require some additional coding or 
modification of code. 

The Blended Model: A Holistic Approach to Creating and Evaluating Web-based Learning 

Environments 

Each of the perspectives brings a unique point-of-view to the evaluation of Web-based training and other 
educational environments. Blending the three approaches helps to ensure a holistic assessment of content from both 
learner and organizational viewpoints, and of the learning environment from learner, instructor and organizational 
viewpoints. This blended approach considers professional standards from each subject matter expertise - 
educational, psychological and technological - during the evaluation of Web-based training. 

The educational and technological perspectives prescribe a process for creating online learning that ideally 
includes evaluation throughout creation, while the psychological perspective focuses only on evaluation after 
implementation or delivery of the training. The educational and psychological perspectives include a substantial 
amount of evaluation to assess the effectiveness of the training, and to obtain suggestions for modifications prior to 
the next implementation of the Web-based training. 

Uniquely, the educational perspective is the only one that ensures that training content is accurate; the 
psychological perspective is the only one that measures the long-term affects of the training on the learner and the 
organization; and the technological perspective is the only one that ensures accurate functionality of the training 
system. While the educational and psychological perspectives are concerned with the front-end or what the learners, 
instructors and others see, the technological perspective is concerned more with the back-end or what supports the 
training on the Web. The technological perspective only considers the front-end insofar as the interface adheres to 
user- and usage-centered interface design principles, which will help ensure the learner can focus on the content to 
be learned rather than becoming distracted by a poorly designed system. This does not mean that the learner will not 
have to learn to use the system, but ensures that the system supports the learning and does not artificially interfere 
with learning. 

What makes the blended approach more holistic is that all elements of the training system that affect the 
learner’s experience are considered in order to make improvements - the content, cognitive level of learning of the 
content and learning exercises, appropriateness of activities and assessments for the learner, and the suitability of the 
interface and system to support learning in accord with hardware and software constraints. We propose that this 
blended model not only combines perspectives but also refines them into five phases as shown in Figure 3, with 
evaluation interwoven throughout much of the process as explained in Table 1. The model also depicts the three 
perspectives (which helps depict their influence on the blended approach) and briefly describes the focus of each 
phase as defined by the blended approach. 
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Figure 3. The Blended Model with a focus on evaluation. 
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T able 1 . Phases of the Blended Model. 
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The Analysis and Requirements Gathering phase begins with a thorough analysis of the performance-gap 
or knowledge-gap experienced by the organization. The purpose of this analysis is to ensure that training is the best 
solution to the organization’s problem by identifying the challenges and examining potential solutions to problems 
from an organizational, stakeholder (e.g., supervisors) and learner viewpoint. Then, a determination of whether the 
Web-based learning is the best solution is explored by examining the number of learners who need the training, the 
technology available at the hosting facility and learner location, and the competency level of the learners. Once it is 
determined that Web-based training is the best solution, an overall goal for the training is established. 

Also in this phase, the parameters of the training are defined, including the content to be covered, level of 
learning that is needed (e.g., from memorization of facts to gaining problem-solving capabilities in the content 
domain), specific target audience who will participate in the learning, technological capabilities of the hosting 
facility (e.g., the organization), and the number of learners to be trained. The attributes of the learners are also 
gathered including, prerequisite knowledge (what they already know that will affect the training content), 
technological capabilities of the system(s) they will use to participate in the training (e.g., at home, in the office), 
and proficiency with the use of the Web and Web-based training/learning environments. 

In the Planning and Design phase, objectives (from the vantage-point of the learner) are written that 
support the learning. For example, a learning objective for a course that attempts to teach at the problem-solving 
level of cognition might read, “At the completion of the training, learners will be able to evaluate a proposal for 
thoroughness, completeness, accuracy and cost.” The appropriate mode of the Web-based instruction - self-paced 
or instructor facilitated or a combination - is also determined. If appropriate, content is broken into “chunks” or 
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lessons, which will in part dictate how development would proceed. Instructional strategies and learner activities 
that support the learning within that mode are then selected - these can be activities that the learner can complete on 
their own, with other learners online, with other learners in a face-to-face exchange and so on. Determinations are 
made regarding the inclusion of multi-media (e.g., sound files, videos) and tools (e.g., discussion boards). 

Technology that supports the learning and is within the budgetary constraints of the organization and 
stakeholders is selected. These include databases to hold data submitted by learners, interface design, software and 
hardware needed. If the training is meant to use an existing learning management system or course management 
system, the system(s) is (are) examined to determine if modifications are needed to support learning. 

Planning for the content and technology to be used based on the capabilities of the host system and the 
learners is critical prior to design. This plan and design establishes the constraints of the system, and therefore, the 
constraints on the learning content and activities that are created in the development phase. It is during this phase 
that formative evaluation begins. Outside evaluators (subject matter experts such as programmers, systems 
designers, interface designers, content experts) review the plan and design for adherence to the requirements 
gathered prior to the development of the system, and alignment with professional standards. In particular, the plan 
for training design is reviewed to ensure that learning activities: support the desired cognitive levels of learning 
(indicated in the learning objectives), can be accomplished with the technology desired for the activity and can be 
implemented given the constraints specified in the analysis and requirements gathering phase. 

The Coding and Development phase begins the implementation of the plan and design created in the 
previous phase. Development of the content and activities begins simultaneous with the creation of the back-end 
(e.g., databases) and front-end (e.g., interface) of the learning environment. 

Formative evaluation is also conducted, and occurs at multiple levels. For example, as the content and 
activities to support a lesson are created (and prior to their implementation on the Web) subject matters experts 
review them for clarity and alignment with the cognitive levels of learning for the training. Then, the lesson is built 
on the Web and evaluated again by experts knowledgeable in interface design, programming and systems design. 

As additional lessons are created and implemented on the Web, they are again evaluated by subject matter experts 
and experts knowledgeable in interface design, programming and systems design to ensure congruence of the 
navigation and usability of the interface. Prior to implementation, the completed training program is evaluated by 
the client and/or stakeholders, subject matter experts, systems designers and programmers to ensure that the training 
is cohesive - including the content, activities, navigation, interface(s), database(s) and other systems. 

In the Release and Implementation phase, the training is actually delivered and implemented with the 
learners. Depending on the design, the instruction can be either led by instructors, or completed independently by 
learners. In addition, subject matters experts and experts knowledgeable in interface design, programming and 
systems design can “lurk” during the administration of the training, or observe the implementation of the training by 
reviewing virtual classroom sessions, online discussions, assignments submitted by learners, and other training 
artifacts. 

In the Summative Evaluation in the Near-Term phase, the training experience is evaluated immediately 
after the training is completed. This means that assessments to measure student learning are administered in the 
learning environment, and learners evaluate their experiences in the learning environment. If training is conducted 
over an extended period of time, near-term evaluation can be conducted after each lesson so that problems and 
challenges can be addressed before completion of the training. 

In this Summative Evaluation in the Far-Term phase, the long-term effects of the training are assessed. For 
example, change in learner performance as related to, and as a result of, participation in the training can be assessed 
through performance evaluations completed by supervisors or through individual job productivity. The affect on the 
organization can be measured by factors such as changes in morale, turnover, absenteeism, and safety violations as 
they relate to the training. 

The blended model is a way of bringing together proven approaches from fields related to Web-based 
training design to influence the creation and evaluation of training so that the training supports learning and contains 
the capabilities needed by organizations. In conducting evaluation using this approach, an instrument should reflect 
the blending of the approaches such that all aspects of the training are considered. 

The Blended Model of Evaluation 

The blended model of evaluation combines the evaluation approaches from three traditionally unrelated 
fields into one holistic evaluation method for web-based training. Using this model, evaluation data can be collected 
from the training team, learners, supervisors and organizations. The training team consists of a training designer in 
charge of developing the training, other team members who are designing and developing content including, other 
training designers, systems designers, programmers, interface designers, graphics designers, and content-subject 
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matter experts. Data can be collected from learners as they proceed through and complete training or during 
development from individuals similar to those who will eventually be the learners. Finally, some time after training 
was implemented, data can be collected from supervisors of the learners and from organizations to which the 
learners belong. 

Tables 2 and 3 provide example evaluation questions blended from each of the three perspectives. In the far 
left hand column of this exhibit is an indicator of who data would be collected from in order to answer each 
question, while the columns suggest the type of information gathered by each person at each phase of the blended 
model. After each example measure is an indication of its primary influence - educational (EP), psychological (PP) 
or technological (TP). The perspectives can be combined and questions usually asked in one type of evaluation can 
be improved upon through the influence of another type of evaluation. Because evaluation refers to the assessment 
of an existing or planned training Web site (once analysis and requirements gathering has determined that training is 
the best solution to identified problems), evaluation in the blended model begins in the design phase. 

In the blended model, evaluation occurs throughout the development process. Feedback should be obtained 
early in each phase and throughout the phase. For example, after a design for a portion of the training Web site is 
establish, it can be sent out to learners for review as the design for other areas of the Web site continues. In the 
design phase, the training designers select learner activities that are appropriate for the Web-based delivery medium, 
reflect the goals of the organization and allow learners to build the knowledge and skills they need to improve 
performance. In this phase of evaluation, information can be collected from the training team, learners and 
supervisors. The training team answers questions about the scope of the web-site designs, the match between the 
training goals and objectives, the appropriateness of the training goals, training content, and the completeness of the 
web-site design, while learners (or individuals similar to those who will eventually be learners) provide feedback on 
the sufficiency of information, the accessibility of documents, and the instructional activities. As additional portions 
of the training are developed, evaluation should continue. 

During the development phase the training team, learners and supervisors assess the content and 
presentation methods, the relationship of the content to the training goals, the source code and web-site functioning 
Next, during the implementation and delivery phases, the training team, learners and supervisors answer questions 
about issues such as the continuity between the objectives and the outcomes of training, the appropriateness of the 
goals, the efficiency of the web-site. 

At the conclusion of training, near-term and far-term evaluations are conducted. For near-term evaluations, 
the training team and learners answer questions about whether the training goals were achieved and if the training 
should be modified. Fearners can provide information at this phase about how much they actually learned, and 
whether the learning activities facilitated their learning. In far-term evaluation, supervisors can provide information 
about the extent to which the training impacted job performance. In this phase data can also be collected from 
organizations to determine if the training had an impact on the organizational level issues such as morale, turnover, 
and accident rates. 

By drawing on the educations, psychological and technological perspectives, the blended approach to 
evaluation can provide more informative and accurate information about a Web-based training program than any 
individual perspective. Combining these approaches can help to ensure that the quality of Web-based training in 
maximized. 
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Table 3. Example evaluation measures using aBlendedModel approach. Focus on information the Learners, 
Supervisors and'or Organization provide. 
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Novice Designers and Real World Problems 
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Abstract 

The purpose of this study is to reveal novice designers’ problems within which they are in the process of 
designing constructivist learning environments for formal school settings. Creating real world mathematics and 
science problems and embedding them into courses are a challenge for novice designers. Qualitative research 
method will be used to collect and analyze data for this study. Also, this study will propose recommendations on 
how to improve the training of novice educational program designers. 

Ai one of the most recent developments in mathematics and science learning, embedded data design stands 
to offer reasoned decision making and motivation on students’ transfer abilities. Also, problem complexity helps 
students deal with real world complexity and gain confidence in their beliefs in constructivist learning 
environments. Many researchers argue that when the curricula organized around authentic problems create more 
experiences that the students faced and learned informally in their daily life (e.g. The Cognition and Technology 
Group at Vanderbilt 1990; Brown, Collins, & Duguid, 1989). However, despite the continued emphasis on 
constructivism all over the world, it’s still hard to find good examples of constructivist learning environments in 
Turkish Primary and Secondary School System on mathematics and science learning. Perhaps there is a lack of 
theoretical knowledge among designers or teachers and/or they may be having some problems during the 
development process. 



Method 

This study attempted to examine novice designers’ problems when faced with the task of designing 
constructivist learning environments on mathematics and science for primary and secondary school curricula. At 
Osmangazi University in Eskisehir, Turkey, School of Education offers an elective course called New Approaches 
in Learning. Every fall and spring semester approximately 15 4* year (seniors) undergraduate mathematics and 
science students take this course as an elective. At the end of the course, each team (consisting of 3 or 5 students) 
has to develop a two or three hour program (for math and science courses) to facilitate learning for primary and 
secondary school students. Those programs are directed to support formal school courses and bring constructivist 
thinking for designing courses and motivating students. These senior students will be a formal school teacher next 
year in Turkish Educational System therefore, at least they have to have an idea about designing a constructivist 
learning environment. 

The research project examines The Casper Project which was developed by The Cognition & Technology 
Group at Vanderbild (1997). Eor the project, students take real world mathematics examples from The Casper 
Project to develop their own problems. After developing their projects, the students tried the program out to see if it 
matched the constructivist learning environments’ characteristics. 

Hoepfl states that (1997, http://scholar.lib.vt.edu/eiournals/JTE/v9nl/hoepfl.html ) the ability of qualitative 
data to more fully describe a phenomenon is an important consideration not only from the researcher’s perspective, 
but from the reader’s perspective as well. "If you want people to understand better than they otherwise might, 
provide them information in the form in which they usually experience it" ( Lincoln and Guba, 1985, p. 120) . 
Qualitative research reports, typically rich with detail and insights into participants’ experiences of the world, "may 
be epistemologically in harmony with the reader’s experience" ( Stake, 1978, p. 5 ) and thus more meaningful. 
According to those explanations of qualitative research, the researcher used this method to gather rich and detailed 
data for this study. 

1 1 students in the course participated in the study. Interviews with the students, questionnaire and the 
online discussion group were being used to gather data for this study. Interviews with the students took place all 
sections of the course such as in the beginning, middle, and the end. In the beginning the instructor and the students 
discussed the real world problems and The Casper Project conversed about their usage for designing instructional 
program. After beginning the developing process the conversation continued between the instructor and students. 
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At the end of the course the in depth dialogues were analyzed using a content analysis technique. All of the 
conversation process with the students were collected and coded for analysis. 

To collect quantitative data from the students, a questionnaire that consists of 20 questions was given to 
the students at the end of the course. The questionnaire contained likert types of questions about the constructivist 
learning guidelines, real world problems and their usability in formal school setting. The researcher, according to 
the feedback of the students, was developed this data form during the developing process. 

The third data collection tool was analysis of an online discussion group. The online discussion group 
consisted of the students who were taken the course and willing to participate to the study. All conversations, chats, 
and e-mails among them were saved to a computer with the intention that they could be used for elaborative data for 
the research. 

Analysis of the data began with identification of the themes emerging from the raw data. During this 
coding period the researcher was identified and named the conceptual categories into which the phenomena 
observed would be grouped. As the raw data were broken down into manageable chunks, the researcher was coded 
the data and translated it into the story line. The results of this research indicated an in depth analysis of the 
strengths and pitfalls that students encounter when modeling The Casper Project Problems in a prescribed setting to 
design constructivist learning environments for formal schools’ courses. 

Results and Discussion 

In order to create a constructivist, in this study a realistic context that is appealing and meaningful to 
students, learning environment for the novice designers was a big challenge. The researcher and the novice 
designers chose Jonassen’s characteristics of meaningful learning to create constructivist learning environments 
(Jonassen, 2002, http://www.coe.missouri.edu/~ionassen/courses/CLE/index.html) . According to these 
characteristics, the results are listed below: 

Active: In order for the learners’ activity, novice designers were embedded some cases into the program. The cases 
were designed with the collaboration of mathematics and science practitioners and two full mathematics and science 
professors. The designers, practitioners and professors were worked together as a development team for the 
projects. The most discrepancies for the teams were developing the real life situations in the cases and embedded 
them into the programs. 

Constructive: Learners integrate new ideas with prior knowledge in order to make sense or make meaning or 
reconcile a discrepancy, curiosity, or puzzlement. They construct their own meaning for the phenomena (Jonassen, 
2002). Knowing what the students knew was very painful process for the designers. To overcome for this problem, 
the novice designers were to work with the target group students during the whole design process. 

Collaborative: Collaborative cases were helped the designers to built collaboration among the students. 

Intentional: Learners were encouraged with highly motivated goals such as after completing this program you will 
be able to solve mathematics and science problems easily, also could transfer your knowledge into your daily life 
after the program. It was not easy for the designer to indicate this goal for these level students. Therefore, the 
designers had to produce more real examples for the learners. 

Complex: Creating ill structured and a complex learning environment was too difficult for the team. For this 
reason, all the members of the team were studied in the traditional system. In this system, all refined and 
oversimplified knowledge were given to them in their all school life. Therefore, they had difficulty to build a 
complex, ill structured and anchored learning environment. 

Contextual: A great deal of recent research has shown that learning tasks that are situated in some meaningful real 
world task or simulated in some case-based or problem based learning environment are not only better understood, 
but also are more consistently transferred to new situation (Jonassen, 2002). In the cases, real life contexts were 
developed by the teams with the help of the learners. 

Conversational: The cases in the program were designed to make them collaborate with each other. 
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Reflective: Cases and examples were connected to another situation with which learners can articulate what they are 
doing, the decision they make, the strategies they use, and the answers that they found. 

Designing real world problems seems to be too hard for novice designers. Creating a good designing 
team, and conversation in the all designing process with the novice designers could be helped them for better 
understanding to design and develop a constructivist learning environment. 
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Design, Implementation and Evaluation of 
An Online Faculty Development Program 

Cengiz Hakan Aydin 
The University of New Mexico 

Abstract 

The Anadolu University has established a center to help its faculty develop their pedagogical skills. The 
Educational Communications and Technology (EIT) Center has planned an online faculty development program in 
2001 but was able to launch it during the spring 2003. Although 36 faculties enrolled the program only 12 of them 
completed. During this one month long structured completely online program, the participants took three online 
courses each of which had four modules. Although the evaluations of the participants, the facilitators, and the 
administrators of the University were in general satisfactory, the EIT and the program development team members 
did not find it successful. However, this first online faculty development initiative has provided significant lessons to 
the members of the EIT Center and the development team as well as to others. Among these lessons, blending the 
online activities with face-to face interactions, creating an electronic performance support system that works as a 
part of an online learning community and motivation of the facilitators were noteworthy. 

Introduction 

Faculty development had been often viewed as a secondary priority before the advances in human 
resources development field. Practices in business have shown that organizations that invested on development of 
their human resources as well as other assets gained a better place and stayed longer in the market. So, most of the 
organizations have started to focus on development of their human resources since early 80s. As it has always been 
the case, the education field adapted this innovation quite sometime later than other fields. Despite this late 
adaptation, many higher education institutions have been investing on creating opportunities for the human 
resources, especially for their faculty, to develop themselves for some years. 

Advances in information and communication technologies have provided universities to carry their faculty 
development programs from face-to-face to distance learning environments. Kelley (2002) indicated that because 
time is a precious commodity at higher education institutions and most of traditional faculty development initiatives 
require faculty to be bound to a specific time and space, the ability to have face-to-face training for faculty is 
virtually impossible. So, use information and communication technologies such as computers and Internet to create 
environments that could eliminate time and space barriers should be employed in faculty development programs. 
Today many universities such as Arizona State University provide online training to their faculty. 

In Turkey, on the other hand, the idea of organized and programmed faculty development is quite new. 
After earning their PhD degrees (most of the times in the process of earning their degrees), the graduate assistants 
immediately start to teach. Usually they learn how to teach through either trial-and-error or observing elderly 
professors. This situation directly influences the quality of instruction in the universities. In order to solve this 
problem, one of the serious steps recently taken by the Higher Education Council (YOK - Yuksek Ogretim Kurulu), 
a government agent that organizes, monitors, and takes most of the decisions about higher education was requiring 
all the doctorate degree students -except the ones in the field of education- to take two standardized non-credit 
courses about instructional planning and assessment. The institutes of educational sciences in the universities have 
been assigned to offer those courses that aimed to help future faculty members gain the teaching skills. Although no 
study has been conducted to assess the effectiveness and efficiency of those courses, in informal conversations most 
of the students expressed their dissatisfaction with these courses. 

Beside these mandatory courses for the future faculty, there are just a few plans or attempts to support the 
current faculty in Turkish universities. One of those attempts was the faculty development program of the Anadolu 
University. During 1999-2000 academic year, Anadolu University started an training program for its faculty. For the 
first year it was planned to help 185 assistant professors improve their pedagogical skills. The assistant professors 
were divided into groups. Each contained 25 people. They were asked to take 9 courses about different aspects of 
teaching such as assessment, academic consulting, and classroom management, so forth. It was a two week program 
and all the participants were required to join the classes 8 hours in a day. 

Observations on the program revealed that attendance requirement and condense design caused high drop 
out rate. Almost 50% of the intended faculty did not complete the program. The ones who were able to continue the 



27 




program also expressed that the full-time attendance requirement and condense structure had lessened the 
effectiveness, efficiency and appeal of the program. 

Therefore, the University decided to offer this program online in 2001-2002 academic year. In addition, it 
was offered to establish a new center for a better organization of such activities as faculty, stuff and/or student 
support programs. As a consequence, the Educational Communications and Technology Center (EIT - Egitim 
Iletisimi ve Teknolojisi Birimi) was lunched in 2001. Its first activity was to design, develop, implement and 
evaluate an online faculty development program for helping the Anadolu University faculty members improve their 
pedagogical skills. 

Eor this activity, three types of online support have been planned: information desk, development program, 
and online learning community. The information desk can be considered as an online faculty development handbook 
that consists of online articles about different aspects of university teaching and links to the online resources. 
Development program involves a structured instruction program that requires all the faculty members -especially the 
assistant professors- participate several short-term online courses. And the online learning community is an online 
communication environment, open to all the faculty members, where they can come together and share their 
experiences, ideas, resources, so forth. 

Due to several reasons such as lack of technological infrastructure and support from the faculty, the project 
could not started in 2001-2002 academic year as well as the beginning of 2002-2003. However, during the fall 
semester of 2002-2003, one type of support system, development program, has been designed, developed and 
implemented in the spring 2003. 



Design Considerations 

This part of the paper gives details about principles that are taken into consideration while designing the 
online faculty development program. 

In the literature there are several works on principles for effective faculty development programs. Eor 
instance, Murray (1999) pointed out that an effective faculty development program must (1) have institutional 
support, (2) be formalized, structured, and goal oriented, (3) provide a connection between faculty development and 
reward (e.g. academic promotion, support) structure, (4) promote ownership of faculty, and (5) create a belief that 
good teaching is valued by administrators. Steinert (2000), similarly, listed eight principles that guide an effective 
program: 

• Understand and work within institution’s context/culture 

• Ensure that programs and activities are based on needs 

• Promote, buy, and market effectively 

• Offer diverse program and methods 

• Incorporate adult learning principles and theoretical framework 

• Remain relevant and practical 

• Work to overcome common problems 

• Evaluate and demonstrate effectiveness 

Kelley (2002) indicated that although Steinert’ s principles were for a comprehensive faculty development 
program they do not distinguish face-to-face and online faculty development, and can be used to establish an online 
training program for faculty. So, these principles provided a framework for designing the Anadolu University online 
faculty development program. 

In Turkey, presidents of universities have big influences on administrations. The president of Anadolu 
University has always been in favor of using technology into not only administrative but also instructional 
processes. So he accepted and supported the program of creating an online faculty development program at the first 
presentation of the program proposal. This support help designers overcome many operational problems and make 
the decision-taking process easy. In addition, the program presented to the deans of colleges and the directors of the 
departments to gain their support. Although most of them liked the idea, some mentioned their concern about 
technological infrastructure of the University. The president assured those deans and directors about solving the 
technological problems. 

The main goal of the program was established as to improve the quality of instruction and as to extend the 
use of technology in learning and teaching processes by using technology to provide opportunities the faculty to 
develop their knowledge and skills. It was intended to create a program structure that provides participants a flexible 
learning environment to attain this goal. 

A program development team that consisted of the members of the Educational Communications and 
Technology Center (EIT) and several members of the College of Education who worked during design of the 
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previous face-to-face faculty training formed. At the end of needs analysis, the team has chosen four topics to be 
included into the program: (1) Course design, (2) use of educational media, (3) assessment and evaluation, and (4) 
academic counseling. Later, the team decided to design and develop three online courses covering these topics. The 
courses were titled as AGPOl Designing Learning Environments, AGP02 Interactive Instructional Media, and 
AGP03 Evaluation and Counseling. In addition, for each course a team that was responsible of designing, 
developing and implementing the course was also established. Mainly a content expert, several graduate assistants 
and instructional designers formed each course team. The program development team members also worked with 
these teams. 

Additionally the program development team provided some already set standard organizational and 
instructional guidelines to the course teams. These guidelines were created in the light of Murray’s and Steinert’s 
explanations mentioned above. These guidelines were: 

• The number of participants in a course should be 10-20 to provide a better individual and social learning 
environment. 

• The maximum length of the course must be 4 weeks. The best times of the year for such an initiative to 
provide a satisfactory number of participation are October-December in the fall and March-May in the 
spring. So, each course should finish in a month and in a semester each course should be repeated three 
times. 

• The faculty should be encouraged to take one course at a time. So that they will not overwhelmed with the 
requirements of the program. 

• Each course should be designed into modules -one for each week- and requires only half or one hour of 
study everyday. Participants should be able to log on the courses anytime they want in a day. In order to 
provide this flexibility the asynchronous interactions should be preferred rather than synchronous ones. 

• Since the program targets adults and professors, the language used during the program -even in the 
announcements- should properly be used. So, it was decided to use term “activity” instead of “course”, 
never use “student” rather prefer “participant”, choose “facilitator” instead of “instructor”. 

• It was concerned that some of the faculty might feel uncomfortable for taking a course from a faculty that 
has a lower academic title or be a participant in a course with faculty that has lower and/or higher academic 
titles. So, names and titles of the facilitators should be kept hidden in announcements and the facilitators 
should not to use their real names but nicknames. Also, opportunity to have and use a nickname should be 
provided to participants. So, not only participants should not know names and titles of the facilitators but 
also the facilitators should learn whom they are interacting with. 

• The facilitators should provide detailed and enough directions on what they are expecting from participants 
to do. If it is necessary in daily bases, these directions should be reminded to the participants. 

• Each participant may not have an advance computer at home or at office. So, the large files of documents, 
video or graphic elements, programs should be sent to the participants either as hard copy or electronic file 
on a CD via mail. 

• Courses should support different types of interactions such as participant-to-participant and participant-to- 
facilitator, participant-to-content. 

• Social interactions should also be encouraged among the participants and also between facilitators and 
participants. 

• No exam should be administered during or at the end of courses to assess the achievement. Participants’ 
performance should be assessed with their participation to the activities and their products (e.g. syllabi, 
PowerPoint presentations, reaction papers, etc.). Besides, a standard satisfaction questionnaire should be 
used by the program development team to get feedback from participants. 

• Immediate feedback should also be provided to participants. 

• Eor technological and administrative problems, participants should be directed to contact with the program 
coordinator who is actually a member of the development team member. 

At the beginning, some members of the Center and some administrators of the University were in favor of 
requiring a mandatory participation due to concern that not enough number of faculty might want to join the 
program. However, volunteer participation was accepted because willingness seemed more proper to the nature of 
faculty development. 

On the other hand, in order to encourage the participation, it was decided to give a participation certificate 
to every faculty who complete most of the requirements of the courses. The facilitators were required to decide the 
participants who should get the certificate. The certificates can be used for academic promotions. 
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Implementation 

This part of the paper supplies information on the design, development and implementation processes of 
the online faculty development program. This part also provides an insight about what kinds of problems the 
program development team members had to deal with during these processes. 

Ones the standards were set, the course design teams started to work. The biggest challenge faced was to 
convince the facilitators that no face-to-face meeting was necessary in this kind of a program. Most of the 
facilitators had no prior online learning and teaching experience. Even one of them indicated that he had no 
computer skills at all. So, they were a little bit insisted that at least at the beginning or end some sorts of in-class 
activities should be conducted. The program development team persisted that the program should be completely 
online and spent quite a time on training the facilitators on online education. Members of the team attended most of 
the course design team meetings and worked with the facilitators and other members of the course design teams. 

Another problem encountered by the program development team was about ownership of the courses. 
Although the course design teams were often asked help from the program development team, they were, especially 
at the beginning, hesitant about interventions of the development team members. They wanted to own the course. 

So, they were assured that they are the owner of the courses and the development team members were there nothing 
else but to help them. The members were also advised not to intervene too much what the facilitators want to do 
during the course. 

Some technological problems were also experienced during the development of the courses. The WebCT 
was chosen to present the materials and to manage online interactions because Anadolu University had already 
bought the license of this platform. The designers had few experience about this platform and also web authoring; 
and, they could not get any technological support and had to learn and develop most of the materials. 

The design and development of the course materials took almost two months in the fall 2002. Since the 
spring semester in the University starts at the beginning of March 2003 and a time needed to the announcements, it 
was planned to launch the program in April 2003. 

A web site was developed and published on the University web site. It was assumed that especially at the 
beginning of semesters faculty check the University web site often then other times, and while checking they can 
easily see and access the faculty development program site. So, they can find out the announcements. This 
assumption turned out wrong because just a few faculties visited the program web site and only two of them applied 
to enroll the courses. 

After this unsuccessful attempt a new strategy was agreed to market the program and a new time schedule 
was created. Therefore, an email message consisted of announcement and information on the program sent to every 
faculty in the University. As a result of emailing, around 40 faculties applied to enroll the courses that were re- 
scheduled to kick of on May 2003. In was interesting that there were also 5 applications from other universities and 
3 from private institutions beside the Anadolu University faculties. The program development team informed the 
outside the University applicants that this program was sort of an in-service training activity but in the near future 
we would like to open it to the outside applicants, too. Then, the team evaluated the applications and registered 12 
faculties in each course. The other applications could not been included the program because they were new 
graduate assistants. Also, it was thought that it could be better to keep the number of participants as possible as few 
since this will be the first online teaching experience of the facilitators. 

The participants were informed via email on how to register the courses and create nicknames in WebCT 
environment. A help line was also supplied to assist them via phone during the registration as well as during the 
whole program. 

Only 17 faculties signed up at the end of registration deadline. The program development team phoned the 
ones who did not enroll whether they had a problem or not. Most of them asked for some more time to register while 
other indicated they were not able to attend due to several reasons not related to the program but to several other 
reasons such as heavy work load. Although some extra time was allowed to the ones needed only one of them signed 
up in the second week of the program implementation. This late participation was also accepted. So the program 
started with 18 participants. 

However, along the way some of the participants dropped out and only 12 of them completed the courses. 
The ones who dropped out the program asked to indicate the drop-out reasons via email and phone couple times by 
both the facilitators and the program development team members. No respond received. 

The participants had chance to use a nickname instead of their names. However only one of them chose to 
hide his name, others used their own names. 

Although the facilitators were advised to include social and task-oriented group activities into their courses, 
the courses consisted of quite traditional methods. During the courses participants were asked to read the materials, 
do the practices, and answer the questions individually. In general, the course materials were basically static web 
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pages and email tool. One of the instructors found sending hard copies of the reading materials easier than scanning 
and publishing on WebCT. Another one developed a multimedia program and saved on CDs. She also put the 
readings on these CDs and sent the participants via regular mail. The participants who were at the same campus had 
not problem in terms of receiving the mailed materials but there were couple participants from other towns who got 
the materials a week after the courses kicked off. Email was the mail communication tool in the courses. However, 
one of the facilitators tried to conduct a chat session. Only one participant joined this session. 

After the courses completed, the participants were invited to meet with the facilitators and the program 
development team members. Only 7 of them were able to come the meeting during which the president presented 
the certificates to the participants. Moreover, they were asked to indicate freely their thoughts and recommendations 
about the program. 



Evaluation 

Evaluation of the participant faculties, the facilitators, the Anadolu University administrators, and the EIT 
Center and the program development team members have been summarized in this part of the paper. 

A satisfaction questionnaire was developed to administer at the end of the implementation. It included 
items concerning media, method, and content employed into the courses as well as the facilitators. Additionally 
there were items related to general satisfaction for learning online. However, this questionnaire was not administered 
owing to insufficient number of participants. 

On the other hand, as it has been mentioned above, the participants asked to express their ideas and 
recommendations during presentation of the certifications. Additionally, unstructured informal face-to-face 
interviews were conducted with some of the participants. 

Almost all the participant concern and recommendation focused on heavy content of the courses, lack of 
enough directions and interactions. They indicated that it would be more beneficial if the content of the courses were 
lighter, goal-oriented, and require lesser work. They mentioned that sometimes they could not understand what they 
were supposed to do during the courses. 

In addition, the participants indicated that the program and the training procedures were not introduced 
enough and as detailed as it should be. They said that this was one of the main reasons most of their colleagues 
could not attend although they though of. 

The participants, however, found the idea of online faculty development very appealing and indicated that 
they would like to join voluntarily these sorts of activities in the future. 

Besides the participants, the facilitators were also interviewed to learn their satisfaction from the courses. 
One of the main points that they mentioned was they did reach what they were planned to aim. Their main goal was 
to have an online teaching experience. They though that they learnt a lot during the program. They expressed that 
online teaching was not as easy as it seemed and required more work than in-class teaching. They pointed out the 
importance of motivation. They had no financial or any other type of motivation. They said that it would be hard to 
continue the program without any motivation. 

Furthermore, the facilitators agreed on the necessity of integrating a face-to-face component to each course. 
They mainly used the participants’ complains about lack of enough directions and interactions. 

Administration of the Anadolu University, on the other hand, was pleased to be able to complete such a 
program and encouraged the members of the EIT and the faculty development program team to continue working on 
this program. 

From the EIT and the program development team perspective, effectiveness of the program was evaluated 
in the boundaries of the framework provided by Steinert. Steinert’s first guiding principle was to understand and 
work within institution’s context/culture. Several issues, related to the Anadolu University’ culture, were not noticed 
by the program development team and have influenced the effectiveness of the program. One of these was 
associated with the attitudes of the facilitators. Majority of the facilitators have had the idea that face-to-face 
interactions must be secured in this sort of a program in order to provide effectiveness, efficiency and appealing. 
They did not believe that faculty development can be achieved with a completely online program. However, they 
were asked to design their courses completely online. The EIT members thought that this attitude and obligation 
have influenced their performance in the program. In addition, although they have other duties and heavy work load, 
they volunteer to contribute this program because they wanted to have an online teaching experience. But during the 
design and development of the courses they noticed that online teaching required more time and work than face-to- 
face teaching. One of the facilitators even mentioned that if he knew online teaching was this hard he would not 
become a facilitator. The heavy work load of the facilitators and the program development team members have also 
lessen the effectiveness of the program. 
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Moreover, it was assumed that if the system built, the faculty will come. However, they did not. The 
program development team members and the facilitators tried hard to reach the faculty to encourage them to attend 
the program. Most of the ones who enrolled course were later indicated that they wanted to have an idea what an 
online course was. After having gone over the syllabi they had what they wanted and decided not to continue the 
program. This sort of an attitude can also be related to the culture of the organization as well as insufficient needs 
analysis procedure. 

The EIT and the program development team members have also believed that improper marketing strategy 
was affective on low participation rate. In other words, the goal and the process of the program could not been 
introduced to the faculty adequately. 

In sum, the EIT and the program development team members have indicated that the program was actually 
not a successful one in terms of reaching its goals, especially promoting ownership of the faculty and creating a 
belief that good teaching was valued by administrators. However, they have mentioned that they learnt a lot from 
this first experience and developed new strategies and ideas to make the online faculty development program of 
Anadolu University more effective, efficient and appealing. 

Conclusion 

According to Murray’s and Steinert’s principles of effective faculty development program, the online 
faculty development program of the Anadolu University can be considered as unsuccessful. However, some lessons 
have drawn from this experience. 

One of the main lessons is correlated to Kearsley’s (2002, p. 44) words: “I am sure that over time, more 
students, teachers, administrators, institutions/organizations, subjects, and cultures will become amenable to online 
education. But, for the present, it is important to think critically about whether it makes sense to offer certain course 
or program in online form and not to assume that it will work everybody.” In other words, this sort of a program 
should have some face-to-face components because both the faculties and the facilitators are not ready for a 
completely online training in Turkey. But the number of the face-to-face components should reduce every time the 
program repeats. So that the faculties have a chance to feel the online learning and teaching experience in sort of a 
situated learning environment and be able to offer completely online courses and programs that require no time and 
space limitation in higher education institutions. 

Another lesson learnt from this program is that some sort of a “kanban”, or just in time approach might be 
better strategy for online faculty development due to heavy work load of the faculties. Scales (1997) states that 
electronic performance support systems can provide human resources of an organization/institution have the right 
training at the right time and place. Similarly the faculty might benefit from an electronic performance support 
system that aims to provide just-in-time training them. However, current research (e.g. Jones, 1995; Gunawardena, 
1995; Nolla, 2001; Tu & Mclsaac, 2002) have shown the significance of social interactions in learning environment. 
On the bases of these studies, it could be claimed that this sort of an electronic performance support system should 
be supported by creating an online learning community. 

Building an online learning community might also help the designers of the online faculty development 
program create a more flexible environment in which each faculty can find something for her/his individual needs. 
Therefore, an ownership feeling can be promoted. 

Another important lesson learnt is about motivation. Although the program provides some sort of a 
motivation to the participant faculty by the certificates that can be used in academic promotions, similar or different 
type of a motivation should be provided the faculties who help creation of this kind of a program. Especially 
facilitators need the motivation. 

As a conclusion, this first online faculty support program in Turkey might seem not able to reach the goals 
but it provided valuable lessons to developers and coordinators of the program. 
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Abstract 

Research related to multi-channel communications provides very few guidelines to assist instructional 
designers in the implementation of audio in multimedia learning. In recent years, however, research studies 
conducted at the University of California have proposed several principles, some of which might be applicable to 
the implementation of audio in multimedia courseware and e-learning. This paper presents an analysis of current 
practices in commercial e-learning courses (N=18) that incorporate audio as a primary delivery media. The results 
indicate that the principles for temporal contiguity, coherence, and personalization are implemented for the 
majority of courses; principles related to modality and redundancy are not. 

Introduction 

Deciding when, where, and how to incorporate audio into multimedia programs is a very complex issue for 
instructional designers. Research related to theories that focus on multichannel communication and learning theories 
have been inconclusive (Barron, 2003). For example, Lang (1995) conducted an extensive review of the literature on 
multichannel research and examined 22 research studies. Her conclusion was, “Forty years of research has yielded a 
hodgepodge of contradictory conclusions” (p. 86). Moore, Burton, and Myers (2003) agree, stating, “We feel that 
instructional designers looking for simple rationale, methods, or guidelines for effective multimedia (multiple 
channel) presentation will be disappointed in the relevant research” (p. 998). 

Compounding the dilemma is the fact that it is almost impossible to find more than a cursory reference to 
design guidelines for audio in textbooks that focus on the design and development of multimedia (see Lee & Owens, 
2000; Horton, 2000; Kruse & Keil, 2000; Allen, 2003). Moreover, the few guidelines that exist are vague or 
contradictory. For example, Aamtzen (1991) stated, “When speech is used as the mainstream provider of 
information it should be accompanied by that same text on the computer’s screen” (p. 363). However, other authors 
advised designers to “avoid echoing text with audio” (Gibbons & Fairweather, 1998, p. 375) and to describe 
graphics by “words presented in the form of audio narration, not by narration and redundant text” (Clark & Mayer, 
2003, p. 109). Alessi and Trollip (2001) take a compromising approach, stating that the best approach may be to 
“provide controls that encourage learners to use either the text or audio presentation, not both at the same time” (p. 
75). 



Multimedia Principles 

Recognizing the dearth of design guidelines, Richard E. Mayer and his colleagues have recently conducted 
dozens of research studies. These studies were designed to help answer questions, such as: 

• How do people learn from words and pictures? 

• What is the best way to design multimedia messages? 

A systematic summary of the studies culminated in the hook Multimedia Learning (Mayer, 2001), in which 
he states, “My premise in this book is that the answers to these questions require a program of careful, systematic 
research” (p. ix). 

Several principles were derived from the research studies, many of which are applicable to the 
implementation of audio. The principles are a cognitive theory of multimedia learning, which “assumes that the 
human information processing system includes dual channels for visual/pictorial an auditory/verbal processing, that 
each channel has limited capacity for processing, and that active learning entails carrying out a coordinated set of 
cognitive processes during learning” (Mayer, 2001, p. 41). Specifically, the principles and recommendations listed 
in Table 1 were outlined in Multimedia Learning (Mayer, 2001) and e-Learning and the Science of Instruction 
(Clark & Mayer, 2003). 
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Table 1. Principles and recommendations. 



Principle 


Mayer, 2001 


Clark & Mayer, 2003 


Modality 


Students learn better when words in a 
multimedia message are presented as 
spoken text rather than printed text. 


Present words as speech rather than onscreen 
text. 


Redundancy 


Students learn better from animation 
and narration than from animation, 
narration, and text. 


Avoid presenting words as narration and 
identical text in the presence of graphics. 


Temporal 

Contiguity 


Students learn better when 
corresponding words and pictures are 
presented simultaneously rather than 
successively. 


Avoid separating information that must be 
integrated for learning. 


Coherence 


Students learn better when extraneous 
material is excluded rather than 
included. Student learning is hurt 
when interesting but irrelevant sounds 
and music are added to a multimedia 
presentation. 


Avoid e-lessons with extraneous sounds. 


Personalization 


Multimedia messages result in better 
transfer performance (but not 
retention) when the verbal material is 
presented in a conversational style - 
using first and second person. 


Script audio in a conversational style using 
first and second person. 



Method 

The purpose of this study is not to attempt to replicate or validate the research and principles set forth by 
Mayer. Rather, the intent is to examine commercially-available e-learning courses and determine whether or not the 
principles are being reflected in practice. 

Eighteen e-learning companies were selected for the analyses, based on the following factors: 

1 . Prominent e-learning companies as determined by recent articles, publications, and industry analyses, - 
- e.g.. E-learning - Is this the bottom? (Mirus Inc., 2002). 

2. Companies that are marketing custom or off-the-shelf courseware at national and international 
conferences, such as Online Learning and TechLearn. 

3. Companies marketing their online courseware through leading technology magazines, such as Training 
and Learning & Training Innovations. 

4. Companies that provide an online demo or courseware that is available free-of-charge to the public. 

After locating numerous online courses, the following criteria were applied to select the courseware 
included in the analyses: 

1 . Courseware that used audio as a primary delivery media — courses without audio or with short, 
intermittent audio segments were excluded. 

2. Courseware that included at least a major portion of an actual lesson (with functional interactions). 

3. Courseware that was interactive and self-paced. 

4. Courseware that was designed for adults. 

5. If more than one course was available at a particular company, a topic of general interest was selected 
for review. 

Table 2 lists the courses were selected for the analysis: 
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Table 2. Selected companies. 



Company 


Program/Lesson 


Advance Online 


Defensive Driving 


Alamo Learning Systems 


Rational Thinking 


Allen Communication 


Making History 


Applied Learning Labs 


Financial Literacy 


Brightline Compliance 


Preventing Workplace Harassment 


Element K 


PowerPoint 2000 


GoTrain 


DC Circuit Theory 


IBM Mindspan 


Marketing demo 


Imparta 


Marketing demo 


Intercom Training 


Good Grief, Good Grammar 


KnowledgeNet 


Cisco IP Phone 


Lean 


Lean Mfg Concepts and Principles 


Medline Plus 


Anthrax 


NetG 


Microsoft Outlook 2002 


Prime Learning 


Analyze Problems Creatively 


Rosetta Stone 


Spanish: Objects and Relationships 


SkillSoft 


Building Relationships for Continuing Success 


Street Smart Communications 


Ergonomics Training 



Data Collection 

A spreadsheet was created that contained all of the e-leaming lessons and variables for investigation. 
Relevant parameters were defined, and drop-down menus were inserted with the variables. Working independently, 
both authors documented the information for each program. If there was disagreement between the responses, the 
issue was resolved by a third party (an expert in e-learning design). 

The following information was collected: 

1 . Modality of the primary delivery media (narration/text) 

• Full Text/Full Audio 

• Full Text/Partial Audio 

• Partial Text/Full Audio 

• No Text/Full Audio 

2. Redundancy related to animations 

• Animation/Text/Narration 

• Animation/Text 

• Animation/Narration 

• Animation - without Text or Narration 

3. Temporal contiguity for animations/narrations (High, Low) 

4. Presence or absence of sound effects and music 

5. Coherence of audio and animations (High, Low) 

6. Style of language (Formal, Informal) 
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Results 



Modality Principle 

The advice from Clark and Mayer is to “present words as speech rather than onscreen text” (2003, p. 86). 
To investigate the current practice related to this principle, each program was assigned one of the following values, 
based on the combination of instructional text and audio. 

1. Full Text/Full Audio 

2. Full Text/Partial Audio 

3. Partial Text/Full Audio 

4. No Text/Full Audio 

The most common combination for delivery was Full Text/Full Audio — text that was mirrored by verbatim 
audio. Ten of the courses (55%) employed this approach. The second most common format (33%) was Partial 
Text/Full Audio. This approach consists primarily of audio narration with bulleted text. Only one of the programs 
employed the No Text/Full Audio approach advocated by Clark and Mayer (see Figure 1). Another approach (Full 
Text/Partial Audio) was also used in one program. 



Full Text 
Full Audio 
55% 




Partial Text 
Full Audio 
33% 



No Text 
Full Audio 
6 % 



Full Text 
Partial Audio 
6 % 



Figure 1. Implementation of audio. 

Redundancy Principle 

The redundancy principle states, “Students learn better from animation and narration than from animation, 
narration, and text” (Mayer, 2003, p. 147). Nine of the eighteen programs in this study did not contain animation 
sequences. The nine programs with animations were examined relative to the redundancy principle. The following 
options were noted for the animation sequences: 

1. Animation/Narration/Text 

2. Animation/Narration 

3. Animation/Text 

4. Animation - without Text or Narration 

The most common combination for delivery was Animation/Narration/Text. This combination was 
observed in five of the nine programs (56%). The combination recommended by Mayer (Animation/Narration) was 
implemented in 2 of the programs (22%). In addition, one program consisted of Animations, without text or audio, 
and one program combined Animation/Text (see Figure 2). 
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Animation 

Narration 

22 % 



Animation 

11 % 



Animation Text 
11 % 




Animation Text 
Narration 
56% 



Figure 2. Implementation of animation. 

Temporal Contiguity 

Several of the Mayer’s research studies found that learners performed better on retention and transfer tests 
“when corresponding portions of animation and narration were presented simultaneously rather than successively” 
(Mayer, 2003, p. 96). Looking again at the subset of lessons that contained animation sequences, the following 
values were assigned to temporal contiguity. 

• High - most of the animations and narration were presented simultaneously 

• Low - few or none of the animations and narrations were presented simultaneously 

There was only one program that was judged to have low temporal contiguity (Advance Online). This program 
contained animations that played after the narration and introductory text. 

Coherence 

Mayer and his colleagues also found that, “Students learn better when extraneous material is excluded 
rather than included (2001, p. 1 13).” In the auditory realm, this would refer primarily to sound effects or music that 
are not necessary to convey the content.Table 3 provides information about the implementation and coherence of 
music and sound effects in the lessons. 



Table 3. Audio types and coherence levels. 



Company 


Music (Y/N) 


Sound Effect (Y/N) 


Coherence (H/L) 


Advance Online 


N 


Y 


High 


Alamo Learning Systems 


Y 


N 


High 


Allen Communication 


Y 


Y 


Low 


GoTrain 


N 


Y 


High 


IBM Mindspan 


Y 


N 


High 


Imparta 


N 


Y 


High 


Rosetta Stone 


N 


Y 


High 


Street Smart Communications 


Y 


N 


High 



Eight of the programs included music and/or sound effects. The coherence of the lesson was coded as High 
(those without extraneous audio material) or Low (those with extraneous audio material). Note that only one of the 
courses was judged to have low coherence - Allen Communications. 
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Personalization 

Clark and Mayer recommend using conversational (informal) language rather than formal language. One 
aspect of informal style is to use first and second person rather than third person in the text and narration. Using this 
criterion, the audio in each lesson was evaluated to determine whether it was primarily formal or informal. 

The results revealed that only two of the programs used Formal style (GoTrain and MedLine). In the other 
16 programs, use of second person was prominent (see Figure 3). 



Formal 

11 % 




Informal 

89% 



Figure 3. Style of language. 



Limitations of this research 

This research has several limitations. First, the selection of e-learning courses is a convenience sample of 
the lessons available free-of-charge on the Web. It does not represent a comprehensive view of the entire industry, 
nor does it provide a comprehensive view of the companies listed (only one program was analyzed for each 
company). The sample (18) is relatively small; the majority of the e-learning lessons that are available do not meet 
the criteria for inclusion (interactive lessons with audio as a primary delivery media). 

In addition, no attempt was made to locate the designers or to query them as to their familiarity of the 
multimedia principles outlined by Mayer. Therefore it is impossible to analyze the rationale of the designers who 
produced the lessons. Whether their decisions were driven by theory, experience, bandwidth, or intuition cannot be 
determined. 

Discussion 

The purpose of this article is to provide a review of the principles related to the use of audio in multimedia 
programs. Programs developed by leading e-learning companies were reviewed, with regard to their implementation 
(or lack thereof) of the principles. 

The majority of the programs (55%) incorporated a combination of Full Text/Full Audio. This is in 
contradiction to Mayer’s Modality Principle, which advocates using spoken text instead of written text. There are a 
number of possible reasons that Full Text/Full Audio is the most common combination. First, and foremost, is the 
issue of compliance with Section 508 of the Rehabilitation Act of 1998. By providing redundant text and audio, the 
programs automatically comply with the requirement to script all narrations. Another factor may be the desire to 
provide flexibility related to learning styles and time requirements. Several studies have found that Full Audio 
delivery results in longer delivery time because of the inherent pacing of audio and the inability to skim through the 
material (Barron & Kysilka, 1993; Koroghlanian & Sullivan, 2000). With both audio and text options, students can 
decide if they want to listen to the audio or read the text. In fact, several of the programs provided options to turn 
the audio off completely (Advance Online, ElementK, IBM Mindspan, Intercom Training, Medline Plus, NetG, 
Prime Learning, and SkillSoft). 

Only two of the programs examined (Imparta and Allen Communication) adhered to Mayer’ s Redundancy 
Principle, which recommends the combination of Animation and Narration. Most of the programs (56%) included 
both text and narrations with the animations. The predominance of an approach with both text and narrations is 
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probably reflective of the number of programs that mirrored audio with verbatim text. Although it seems difficult to 
read the text, listen to the narration, and watch the animation concurrently, the text may have been included on the 
screen for ADA purposes. Another interesting point about the animations was the lack of learner control -most of 
the animation sequences played automatically, without options other than global screen options such as Next and 
Back. This is contradictory to recommendations by Alessi and Trollip (2001) that state, “Allow the learner to pause, 
continue, repeat, and in some cases control the speed of change” (p. 72). 

In relation to Mayer’s Temporal Contiguity Principle, the programs fared quite well. Eight out of the nine 
programs with animations were rated as “High” for temporal contiguity, indicating that the animations and audio 
were synchronized and easy for students to follow. The availability of tools, such as Flash, with easy-to-implement, 
low bandwidth options for creating and delivering combinations of audio and animations enables designers to 
comply with this principle. 

The Coherence Principle was also reflected in the programs that were reviewed. The programs included 
minimal music and sound effects. With the exception of one program, the few sound effects in the programs were 
short and pertinent to the content (such as the sound of a phone ringing). Although music was included in four of the 
programs, it was primarily used as background music in the introduction, and it did not distract from the content. 

The argument for using formal language (third person) in a program is that it may convey a more serious 
approach to learning. However, Moreno and Mayer, (2000) conducted several research studies that compared a 
personalized version of a program to a formal version. “Overall, participants in the personalized group produced 
between 20 to 46 percent more solutions to transfer problems than the formal group” (Clark & Mayer, 2003, p. 137). 
The analysis of e-learning programs for this research revealed that 16 out of 18 programs used the informal style, 
consisting of first and second person. 



Conclusion 

Does the fact that current e-leaming courseware do not necessarily adhere to Mayer’ s modality and 
redundancy principles mean that these courses are less effective than they could be, or does it mean that these 
principles might not apply to interactive, e-learning courses because of differences in content, pacing, or learner 
abilities? 

There are critics of the principles who note that most of the research studies, upon which the principles are 
based, consisted of extremely short (less than three minutes) segments (Tabbers, Martens, and van Merrienboer 
(2001). Although difficult to measure, most of the e-learning courses on the market appear to be designed in 
modules that range from 15 minutes to an hour in length. It is quite possible that the optimal modality and 
redundancy for retention and transfer differs, based on the length of the module and the length of the animations. 

The research studies by Mayer and his colleagues primarily focused on technical, cause and effect 
explanations of how various devices or processes work - lightning, brakes, mechanical pumps, etc. Although some 
of the studies examined in this review had technical content, many of the lessons focused on “soft” skills, such as 
financial literacy, building relationships, and workplace harassment. 

Another factor that may influence the applicability of the principles is the pacing/control aspects of the 
lesson. Mayer’s studies do not appear to include learner-controlled pacing (as is common in most interactive, e- 
learning lessons). Whereas Mayer’s modules were primarily composed of 90-180 second animation sequences, the 
e-learning lessons consisted of numerous, short segments (10-60 seconds) with learner control. This difference could 
easily vary the impact on the cognitive load required for various media. Tabbers, Martens, and van 
Merrienboer(2001) conducted studies and found that “In the two groups in which the students set the pace of the 
instruction, no modality effect is found at all. Not only do the students in the visual-user group perform almost 
equally well on the transfer test, on the retention test they even outperform the students in the audio-user group” (p. 
1029). Tabbers (2002) recommends, “Only in situations in which time-on-task is a crucial variable and the 
instructions are system-paced based on the pace of the narration, should spoken text be first choice in multimedia 
instruction” (p. 79). 

Learner characteristics may also influence the optimal modality and redundancy in e-learning lessons. 
Mayer (2001) cautions that design principles based on research “must be qualified with respect to different kinds of 
learners. Additional research is needed to pinpoint the role of individual differences in multimedia learning” (p. 

189). For example, some of his studies showed strong multimedia effects for students with low prior knowledge and 
weak effects for high prior knowledge students (Mayer, 1997). Similarly, high spatial ability students were more 
likely to show a greater temporal contiguity effect than low spatial ability learners (Mayer, 2001). Shih and Alessi 
(1996) seem to agree with Mayer about the need for qualifications in media research. They stated, “Media is a tool 
or a learning environment; we educators or researchers must have a better understanding of it before we can wisely 
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use and design it. Therefore, further and multiple dimensions of research on using audio (sound or voice) in 
education and multimedia are suggested” (p. 217). 
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Appendix 

Advance Online - Defensive Driving 

http://www.advanceonline.com/freecoursedemo.htm 
Alamo Learning Systems - Rational Thinking 

http://www.alamols.com/eLearning/web advantedge.htm 
Allen Communications Making History 

http://www.mentergv.com/products/courseware/new demos/RC%20PLP%20Demo/ 
Applied Learning Labs Einancial Literacy 
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http://www.appliedlearninglabs.com/volt/htmlbased/voltsite.html 
Brightline Compliance Preventing Workplace Harassment 

http://www.brightlinecompliance.com/ 

Element K - PowerPoint 2000 

http://www.elementk.com/e-learning/htm/ 

GoTrain Corporation- DC Circuit Theory 
http://www. gotrain.net/demo.htm 
Imparta/iCoach - Marketing demo 

http://www.i-coach.com/(luilw245ac4ofmnx5ihtr02x)/defaultlogin.aspx 
Intercom Training Good Grief, Good Grammar 

http://www.intercomtraining.com/ 

KnowledgeNet - Cisco IP Phone 

http://www.knowledgenet.com/courselibrarv/demosandguidedtours/ciscoshow2.is 
Lean Performance Management 

http://www.leanedu.com/demomenu.cfm 
IBM/Mindspan - Marketing demo 

http://media.lotus.com/learningspace/student/index.htm 
Medline Plus - Anthrax 

http://www.nlm.nih.gov/medlineplus/tutorial.html 
NetG - Microsoft Outlook 2002 

http://www.netg.com/demosanddownloads/ 

Prime Learning - Analyze Problems Creatively 

http://courseware.geolearning.com/index.cfm 
Rosetta Stone - Introductory Objects and Relationships 
http://www.rosettastone.com/ 

SkillSoft - Basic Lirst Aid 

http://www.smartforce.com/corp/marketing/demo/safety.htm 
Street Smart Communications - Ergonomics Training 

http://www.streetsmartcom.com/flash/main.html 
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Metacontrol: Controlling the Audiovisual Enterprise 

Joseph Bocchiaro III 



Introduction 

Universities, corporations, and institutions of all types have been deploying increasingly complex 
audiovisual systems over the last twenty-five years. Audiovisual control systems have evolved throughout this 
period, as the necessity to tie together disparate types of equipment has increased. With the development of 
sophisticated equipment has come the need to cohesively manage these systems within the overall enterprise. This 
paper explores the recent developments in interconnectivity between audiovisual systems, computer network 
hardware and software, control system software, inventory and maintenance, usage tracking, budgeting, and the 
many other advantages of metacontrol. “Metacontrol” is an artificial intelligence (AI) term meaning the “control of 
control systems.” This paper tracks the evolution of these control systems and the gradual convergence with 
Information Technologies (IT) that is necessary for metacontrol to exist. 

The Evolution of Audiovisual Control Systems 

Audiovisual control systems have always served as an accessory to pieces of audiovisual, audio, and video 
equipment. Some of the earliest types of systems controlled one type or model of equipment, such as the “show 
control” or “multimedia” slide projector / audiotape units. Early audiovisual systems were completely customized, 
brute-force assemblies of TTL (transistor to transistor logic) controllers, relays, hardwired switches and lamps, and 
multi-conductor cables. As audiovisual systems grew, it became evident that these labor-intensive configurations 
would have to evolve. 

Stand-Alone Audiovisual Control Systems 

In the mid- 1980’ s, four manufacturers almost simultaneously developed alternatives to the control wiring 
morass that was then the state-of-the-art. YORK CONTROLS, AMX, CRESTRON, and ESR revolutionized the 
industry with digital signal lines, programmability, integrated packaging, and expandability. Three of these 
companies remain as market leaders to this day. Their systems gradually took into account the growing number of 
control interfaces to external equipment that was developing. This included the traditional dry contact closures, RS- 
232C, infrared, and various proprietary signal protocols such as SONY VISCA. 

As it became easier to build these systems, as demand grew, and as the size and complexity expanded, it 
became evident that the control systems would have to develop further. It may seem a surprise to us today that there 
was tremendous opposition to this particular evolution, as this was the first time that both hardware and software 
were necessary to make an audiovisual system operable. There was distrust in microprocessor technology that could 
“crash.” There were already enough points of failure in audiovisual systems at that time (especially the CRT 
projectors!) so there was an unwillingness to take additional risks. However, the promise of cleaner systems was 
there, and many audiovisual integration firms created a new job description at this point, the “control system 
programmer.” 

Proprietary Control System Networks 

The next two, interdependent major developments involved the human interfaces to the control systems, 
and the development of more robust proprietary digital networks. Each manufacturer produced a proprietary digital 
protocol that could link their distributed components such as camera controllers, power switchers, volume controls, 
etc. Most importantly, these networks linked the touch panel interfaces that would become the de rigueur standard 
of the industry. The ability to create systems with multiple, customizable and changeable interfaces, distributed 
throughout a room or building, with simplified cabling, ushered in a new era of audiovisual sophistication. 

Meanwhile, the personal computer industry was commoditizing personal computers, and the cost of 
networking these computers was plummeting. The audiovisual industry remained fixed on a closed-box, 
proprietary-processor/software model. Because the reliability of personal computer operating systems was not 
robust enough for audiovisual systems, this approach worked very well. However, it was destined to change as 
audiovisual systems continued to proliferate in a myriad of markets, and as trust in the “WinTel” (Windows/Intel) 
platform reliability increased. Using a personal computer and an “off-the-shelf’ touchpanel was not a mainstream 
idea yet. Interestingly, control system manufacturers were concerned with controlling personal computers through 
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their mice and keyboard interfaces, but not necessarily interested in being controlled by the computers themselves. 
One exception, however, was the IBM “Interactive Multimedia Classroom,” (IMMC) collaboration, which married 
an OS/2- based personal computer with an FSR control system. The age of computerized multimedia had begun. 




AUDIOVISUAL ROOM 



Figure 1 . Proprietary audiovisual control system 

LAN-Based IP-Enabled 

The audiovisual industry response to consumer demands for more sophisticated home -based audiovisual 
systems has led to a wide variety of competing control solutions. The sheer market size of this industry has also 
brought the attention of the large software, network, and computer manufacturers. Industry consortia and 
proprietary efforts have led to products that are also useful to many other markets. This is particularly true in the 
areas of IP-based audiovisual distribution and distance learning. The forefront of AV/IT convergence lies in this 
area. According to Rashid Skaf of AMX, “It is essential to utilize the available telecommunications equipment. 
This is a market-driven requirement: multiple rooms that need to be connected to each other, share information with 
each other; utilizing the network infrastructure lowers the total cost of ownership.” 

The early efforts at programming for the Internet involved a programming language called JAVA, 
developed by SUN Microsystems. This flexible platform was ideal for the needs of the audiovisual control systems 
designers. However, the lack of computer industry cooperation with the JAVA language produced variations in the 
code, particularly with the Microsoft releases. This caused some unpredictable results in the implementation, and a 
different platform had to be developed. According to Fred Bargetzi of CRESTRON: “The first generation of 
Internet-enabled systems was designed for a ‘cross-platform’ operating system. Our second generation is based on 
ACTIVE X. This allows for numerous ways to interface to our systems: 1) conversion to HTML using ACTIVE X, 
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which will work with a browser such as Internet Explorer (IE), 2) the conversion of web pages directly into 
executable files for speed and security, and 3) control from any industry standard device such as a Pocket PC PDA 
using 802.1 lb wireless, or from ‘web tablets’ such as the Viewsonics Viewpad.” 




Eigure 2. LAN-based/IP-enabled control system 

While the audiovisual industry was transitioning to digital systems, the IT industry was already grappling 
with its own paradigm shift. This was the migration towards creating networks to connect servers with the 
proliferating personal computers. To make matters worse for the network technologists, there were already 
numerous operating systems, and network protocols varied between different manufacturers. This new equipment 
had to work with the legacy equipment, as well. In 1988, the Simple Network Management Protocol (SNMP) was 
developed, and rapidly became the de facto standard for internetwork management. Based on the “manager/agent” 
model, SNMP became the underlying code for the World Wide Web implementation of the Internet. According to 
Rick Egan of SIMTROL, “Browser-based connectivity to audiovisual systems is possible because of our ability to 
utilize data bases for managing system information; this is built with SNMP traps capturing and logging status 
changes.’’ Because each “agent” device requires minimal software, this protocol is referred to as “simple.” The 
“traps,” which are the event notifications from the “agent” to the “manager” are the fundamental means by which 
control and status are communicated through the network. 
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The use of a conveniently-accessed, familiar, ubiquitous interface such as Internet Explorer has opened up 
new possibilities for control systems. It has taken some of the mystique out of the systems, just as it has enabled 
more people to be able to access, program, and use them. However, as the ability to use lower-cost hardware 
components increases, the overall cost of the systems has decreased. As a result, users are now able to deploy more 
systems. This “tail-wagging-the-dog” effect has created an even greater need for connectivity between the systems, 
as audiovisual administrators find they are short-handed in their facilities. 

Converged Audiovisual and Control Systems 

While the audiovisual industry was developing its’ Internet capabilities, the computer industry was at work 
on content delivery over the Internet. Referred to as “rich media content,” this includes streaming audio, streaming 
video with audio, videoconferencing, data collaboration, etc. Finally it was possible to not only control media 
systems through computer networks, but to deliver the content simultaneously. Specialized components for the 
Internet market included sophisticated software CODECs (Coders / Decoders) to allow millions of ordinary 
computers to display video clips and live broadcasts over the Internet. The widespread deployment of broadband 
Internet connections encouraged users to take advantage of this technology. The use of familiar VCR-style control 
interfaces bridged the user interface between the hardware-based audiovisual paradigm and the software-based 
computer paradigm. Why not reverse this concept and use computer conventions for audiovisual hardware? And 
why not use computer hardware as well? This would indeed be the next step. 

Meanwhile, the audiovisual industry was preoccupied by videoconferencing (VC) applications in all 
vertical markets. Sophisticated VC hardware allowed not just one but numerous users to connect and conference 
simultaneously (multipoint conferencing.) Connections through the traditional telephone interfaces evolved to IP 
connectivity, again pointing the way to a converged AV/IT environment. With this widespread use of VC came new 
problems, namely the control of multipoint control units (MCU’s,) the scheduling of the rooms and the conferees, 
and the management and maintenance of the equipment. Low relative cost, disparate locations, and convenience 
encouraged tremendous investment in this technology, and created a new set of challenges for the audiovisual 
industry. 




Figure 3. Converged audiovisual and control system 

At last it was possible to foresee the end of one of the major stumbling blocks of the audiovisual industry; the 
plethora of cables, signal types, protocols, connections, and disparate equipment types. Perhaps the adoption of 
computer network technology could eliminate the labor-intensive aspects of audiovisual integration, while 
leveraging the tremendous volume discounts of computer hardware. Perhaps the single connection would soon be a 
reality, given the promises of virtually unlimited bandwidth over Gigaspeed or better cabling. 

Embedded Audiovisual Control Systems 

The potential size of the audiovisual control system market has not been lost on the rest of the commercial 
and consumer audiovisual manufacturers, or on the personal computer industry. Control systems are now appearing 
everywhere. As examples, the videoconferencing system manufacturers, (particularly Tandberg and Polycom,) offer 
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powerful control software for their CODECs and MCUs (multipoint control units.) Polycom offers CODECs (the 
iPower series) that are actually personal computers that may be used as control system hubs. Tandberg has long 
recognized and allowed for computer connectivity, even providing a VGA interface to their units. Virtually all of 
the consumer audiovisual manufacturers offer integrated remote controls, some that operate many types of 
equipment, not necessarily of their own manufacture. Some even offer touchscreens that rival the comparable 
commercial units of a few years ago. Extron Electronics, the leading audiovisual interfacing, switching, and 
distribution manufacturer, has recently embedded a control system into their switching systems. 

Not to be outdone, AMX, CRESTRON, and ESR now offer audiovisual components that incorporate their 
control systems. Virtually all audio and video components from other manufacturers are now at least digitally 
controlled if not completely digital, and require proprietary control systems to program and operate them. And as 
we will explore below, the computer software and hardware manufacturers are increasingly offering software- 
controlled connectivity to a variety of audiovisual components. Who is not familiar with the DVD and CD-player 
and recorder capabilities of virtually all current personal computers? Clearly the lines are now completely blurred 
between audio, video, computer, and control systems. 




I AUDIOVISUAL ROOM I 

Eigure 4. Embedded audiovisual control system 

Distributed Component Control Systems 

The proliferation of portable, digital media devices such as digital cameras, digital camcorders, MP3 audio 
players, PDA’s (personal digital assistants,) and others, has created a “dymanic” network challenge. The computer 
industry has been working for several years to integrate data transport with a wide variety of control protocols. The 
two market leaders, Microsoft and Sun Microsystems, and an industry consortium called HAVi, have developed 
open-standards solutions to this challenge. With the development has come the recognition that networks are not 
necessarily static any more, meaning that devices may be connected and disconnected frequently, randomly, and 
without the aid or intervention of a network administrator. In addition, these devices may not be computer or 
network devices at all, but may be considered to be “network appliances,” with embedded, dedicated-function 
network interfaces. Already a wide variety of such devices are available, such as those mentioned above, some cell 
phones, wireless networked personal computers, and now audiovisual control systems. 
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The promise of embedded network interfaces and control systems, with unique addresses and 
characteristics, is that devices may be connected, recognize each other, and communicate with each other. This 
becomes increasingly important as networks increase in size and in geography, such as over a WAN (Wide Area 
Network.) Rob Gingell of SUN Microsystems explains their JINI solution to this concept in the Sun publication 
“The Future of Software:” “. . .as you stretch the [network] space in one direction or another you discover that a lot 
of things aren’t working well. The strategy of network transparency (a completely administered, and static network) 
fails with scale; what is effective when most of the parts work most of the time isn’t a good strategy when the parts 
count is so large that something is broken all the time. All the transparency mechanism does is obscure the failure, 
since, after all, that was what it was designed to do; because, of course, the failure won’t be around for long or occur 
very often. That’s one of the motivators behind Jini Network technology. . . It doesn’t assume the network’s going to 
stay up, or that one size fits all answers and implementations for varying bandwidths, or that latencies are suitable. 
Jini technology makes you deal up-front with the network, and, in so doing, become a part of it rather than its 
adjunct. . . . JINI technology [allows] disassociated resources to discover and interact with each other, mostly in the 
form of devices out and about in the world. . . .The technology acts as a vehicle for creating true mammals of the 
network- applications that exploit, rather than simply deal with, the properties of a networked environment.” 




Figure 5. Distributed component audiovisual control system 

In a competitive and similar fashion, Sibylle Haupert of Microsoft describes UPnP: “Universal Plug and 
Play is designed to bring easy-to-use, flexible, standards-based connectivity to ad-hoc or unmanaged networks in the 
home and small businesses. UPnP leverages Internet technologies to enable seamless proximity networking, plus 
control and data transfer among networked devices. Universal Plug and Play uses declarative wire protocols 
expressed in Extensible Markup Language (XML), and communicated via HyperText Transfer Protocol (HTTP). 

For Universal Plug and Play, networking based on Internet Protocols ensures a proven ability to span different 
physical media, to enable multiple-vendor interoperation, and to achieve synergy between the Internet and home and 
office intranets. Universal Plug and Play can also bridge to non-IP networks and protocols such as Simple Control 
Protocol (SCP).” 

To put this into plainer English, the Microsoft White Paper, “Understanding Universal Plug and Play 
(UPnP,)” states “With UPnP, a device can dynamically join a network, obtain an IP address, convey its capabilities, 
and learn about the presence and capabilities of other devices - all automatically; truly enabling zero configuration 
networks. Devices can subsequently communicate with each other directly; thereby further enabling peer to peer 
networking.” 



48 





The HAVi standard has been developed specifically for media-rich content and control of that content over 
various networks. The standard is supported by numerous consumer / commercial audiovisual component 
manufacturers. A primary transport hardware and software protocol called IEEE 1394, or, in Apple Computer’s 
nomenclature, “Eirewire,” or, in SONY’S nomenclature “i.Link,” is used to combine all functionality over one cable. 
According to the HAVi Technical Background briefing, “HAVi provides an environment for audio and video 
devices to interoperate with each other, irrespective of the actual brands or their HAVi implementation. The HAVi 
architecture is open, scaleable in implementation complexity, platform-independent, and language neutral, ie., HAVi 
can be implemented in any programming language and on any CPU or real-time operation system. . . The benefit of a 
network of interoperable devices is that the whole is greater than the sum of all the components.’’ 

The HAVi standard, conceived initially for consumer / residential audiovisual applications, is 
complementary to the JINI and UPnP standards, as well as to Bluetooth, a wireless communications protocol. The 
recognition that the larger computer network industry is already creating a platform for larger-scale implementation 
allows HAVi adopters to concentrate on the smaller, although still complex, control issues. The devices that HAVi 
will control are not limited to audiovisual components, either. Other common household devices such as garage 
doors, thermostats, lighting systems, window shades, etc. are part of the network. 

Basing the HAVi standard on a well-defined and widely accepted platform is an intelligent decision, and is 
indeed complementary to the Ethernet network architecture. The bandwidth of Eirewire, recently expanded in the 
1394b version to 800 Mbits/sec, is designed to eventually increase to 3.2 Gbits/sec. This provides connectivity that 
is just ahead of widespread Ethernet deployment of 10 Gbits/ sec, with eventual 40 Gbits/sec. Einally there is 
sufficient bandwidth for true audiovisual signals, which may require the transfer of real-time computer screens of 
1280 X 1024 with a transfer rate of 414 megabytes/sec at a typical 72 Hz refresh rate. 

None of these developments by leading manufacturers are ignored by the audiovisual industry. In fact, the 
smart companies are exploiting them. Says Bargetzi: “We’re embracing the PC industry: anything that is a 
commodity item- we’re making provisions for these to be used with our systems and software. . . This concept 
(distributed computing) is trying to bridge the world together: thin clients can communicate with others in a 
network.’’ It is difficult to imagine any application with greater demands on a computer network than audiovisual, 
and the audiovisual industry stands to benefit from these advancements, whose development is largely funded by 
others. What that network consists of is open to question as well, as the bandwidth capabilities of Ethernet are 
developed to meet and exceed Eirewire. The convergence on a solution is not to be developed by the audiovisual 
industry, according to Ken Kalinoski of EORGENT: “In the past we talked about the network- our clients have told 
us ‘you have to work within the network I’ve got.’ This is a client-driven marketplace.” 

Metacontrol and the New Capabilities 

Metacontrol, a term adopted from the artificial intelligence field, is used to describe a condition wherein the 
complexities of interacting systems and changing events require an overall control mechanism. The recent advances 
in computerized audiovisual control systems, and the new, specialized audiovisual and control networks described 
above now allow for such overall control. Eurthermore, the convergence of the audiovisual and information 
technology industries has opened up new possibilities for unifying many of the media production, distribution, and 
storage requirements of every corporate, educational, and other institution. As the information explosion continues, 
it becomes more and more important to manage this information, and the means by which it is presented. 

Many of the audiovisual applications important to audiovisual personnel, facilities managers, and IT 
managers are affected and may be enhanced by this new technology. This is important both to institutions with 
limited resources and to those with a large amount of resources. Small institutions may rely on metacontrol to 
manage their shared rooms, bandwidth, and equipment. Larger institutions may use metacontrol for managing their 
considerable inventory with limited staff, perhaps across numerous cities or countries. Institutions with large 
amounts of data compiled in meetings, such as from voting, audience response, or collaboration, may use 
metacontrol to manage this information and organize it for the use of the participants in real-time or in an archive. 

In short, metacontrol holds benefits to any institution, especially those that are growing on a limited budget. 
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Figure 6. Metacontrol system 

The concept of metacontrol is simply the “control of control systems.” Of course it involves the use of 
computer-industry standards and protocols, rather than creating new ones. As such, at its core, metacontrol is an IT 
application. According to Kalinoski: “You have to bring everything back to work with IT, instead of trying to 
circumvent IT.” This concept has grown in response to several industry trends. The first trend was the necessity to 
provide audiovisual professionals with the ability to monitor or alter the status of room-based audiovisual control 
systems, in institutions with many rooms. The next trend was the necessity to manage the audiovisual equipment 
inventories in these institutions. This includes routine maintenance issues, such as lamp changes in projection 
equipment, or the cleaning and lubrication of motorized equipment. As audiovisual equipment manufacturers began 
to incorporate more sophisticated control interfaces in their equipment, it became possible to monitor and control 
many additional features and equipment interactions. A logical extension of this has been to schedule rooms, 
students, instructors, and other resources. 

The real benefits of metacontrol are realized in specific large-scale system features, which lie beyond the 
technical abilities of most users. According to Kalinoski, “Users do not care about all the network issues. 

Everything has to be handled at the user interface: endpoints, gatekeepers, gateways, routers, and switchers.” First, 
the management of sophisticated equipment such as multipoint control units (MCU’s) in corporate, institutional, or 
distance learning applications requires equipment connection, scheduling, bandwidth management, room 
scheduling, and session control. Another application is the scheduling of portable equipment in a multi-room 
environment. Still another is the help-desk functionality of multi-room environments with limited staff. Another is 
the automation of classroom events such as with media retrieval. Most importantly is the invisible, interconnected 
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manner in which metacontrol configures audiovisual networks and makes them available for simplified and 
immediate use. This is of particular importance to wireless network implementations. 

Much of the current emphasis in metacontrol development pertains to the audiovisual systems themselves. 
Rick Egan, of SIMTROL, explains, “Much of our development is focused on creating interfaces and functions that 
IT professionals already are accustomed to. This includes proactive system monitoring, system health and status, 
meantime between failures tracking, component shelf life, and alarms. IT professionals have a firm understanding 
of everything on their network, but when they go into the boardroom they don’t know what’s going on.” Migration 
to metacontrol systems will enable the IT professionals to better integrate audiovisual systems into their universe. 
Egan continues, “The whole control system is network-based, from the boardroom to the NOC (network operation 
center.” It can be seen then that if metacontrol is to be utilized in the future, or planned from the beginning, then 
audiovisual systems should be designed from the beginning to allow for it. 

Metacontrol goes beyond just the ability for remotely-located administrators to manage audiovisual 
systems, however. According to Skaf, “I see the real evolution in the industry is that we’re currently in control, but 
the ultimate goal is true automation: based on rules, not people touching buttons.” The possibilities for this 
technology are boundless. Skaf continues, “Proctor & Gamble have been conducting consumer research into the 
possibilities of touch screens built into household network appliances. This includes grocery lists on touchscreens 
on refrigerators in the kitchen, health and beauty aids in the rest rooms, and home and automobile maintenance in 
the garage.” Once again, audiovisual control manufacturers are looking far past the entertainment and presentation 
technology fields that they thrive in, to the enterprise-wide control of many types of components and systems in a 
multiplicity of environments. 

There are some fascinating examples of the potential for metacontrol. The first one, well within the current 
capabilities of software and hardware, involves multi-city videoconferencing meeting scheduling. A busy executive, 
instead of having an assistant call numerous meeting participants, room scheduling secretaries, catering services, 
audiovisual technicians, IT personnel, videoconference bridge services, etc., uses metacontrol to organize the entire 
event. Using one web-browser interface, the executive selects the required date, time, recipients, and meeting 
requirements. Metacontrol takes over by sending invitation emails via Outlook, booking the meeting rooms, 
configuring the MCU for the multipoint call, sending emails with the refreshment and food requirements to the 
catering services, alerting the audiovisual group of the pending event, and alerting the security department of the 
level of executives attending at each location. Metacontrol, at the time of the meeting, turns on the audiovisual 
systems in each of the videoconferencing rooms, dials or establishes IP connectivity between the CODECs, and 
configures the system for an audioconference. The meeting may now take place. 

Metacontrol has posted the meeting on the master web page for each of the meeting rooms. It has listed the 
meeting on the displays outside each room, and on the master display on the floor of each conference center. During 
the meeting, a network administrator may browse into any room, view the camera signals and the electronic 
presentations, and be available on a help-desk basis. Participants have “help” buttons on their screens to access this 
administrator, who may take control of their system remotely if necessary. But metacontrol does not stop at the end 
of the meeting. When the meeting moderator inputs that the meeting is finished, from the local control system 
interface, metacontrol performs the required cleanup functions. It logs the usage of each room and the equipment 
for maintenance and equipment use tracking purposes, logs the bandwidth, time, and successfulness of the 
videoconference equipment, connections, and MCU, calculates the “charge-back” billing for the call and meeting, 
and submits a report to the accounting department for use in tracking all expenses associated with the meeting. 
Metacontrol has also collected the notes from all of the interactive electronic whiteboards used in the 
videoconference rooms, and emailed the files to all of the participants for their record of the meeting. It has also 
digitally recorded the meeting and saved it on the company file server. 

Another completely different use of metacontrol occurs in the home. The home control system, connected 
to the refrigerator and the pantry, has been logging the food inventory continuously via barcode scanning as items 
are added or removed. The busy executive, upon arriving home, wishes to know what dinner may be prepared with 
the available ingredients. Metacontrol offers a list of the possible recipes, via the touch screen on the refrigerator. 
The executive is also interested in preparing a particular dish for the next day. The portable web browser screen is 
used to browse through recipes, from the comfort of the living room, and when one is found, the available items are 
evaluated. Missing ingredients are ordered through the Internet to a delivery service. The next day, all items are 
delivered, logged, and cooking can begin! 

Obviously the sophistication of these scenarios supposes that extensive programming has been performed 
to set up the possibilities. The final level of metacontrol is one of the final levels of artificial intelligence: that the 
systems may learn from the patterns of behavior of the users, adapt to these behaviors, and modify, predict, and even 
anticipate future behavior. Although we are a way off from these possibilities, it is the groundwork of the systems 
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and program designs of the present, along with the standards being developed today as described above, that are 
paving the way to this fantastic future. 

The promise of metacontrol should be of interest to technology professionals of all types who are interested 
in audiovisual and computer industry trends. Those who have the responsibility of planning, designing, 
implementing, and managing audiovisual systems must understand the ramifications now, as to how the newest 
technologies can be used for operational cost reduction and administrative simplification. The benefits will certainly 
be found in the ability to manage a growing number of systems, with dwindling resources and increasing demands. 
This ease-of-use and satisfaction with the technology will give users confidence and successful audiovisual 
experiences, which ultimately are the return on investment that are required to support metacontrol. 
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The Application of Business Models to Distance Education as a 
Result of Experiences with Interactive Television 

Doris U. Bolliger 
Rich Josephson 
St. Cloud State University 

Abstract 

In this paper, the authors outline the experience of teaching and learning in an interactive television (ITV) 
classroom. Both the instructor and the students experienced significant challenges. The authors relate ITV, thought 
of as a mature technology, to product life cycles and business models. Business models focus on expectations of 
clients, standardization of a process, performance, and efficiency of product delivery. These elements are typically 
present when a product is in the mature stage of the product life cycle. 

Introduction 

Educators have taught courses in interactive television (ITV) classrooms for quite some time. These 
interactive distance learning classrooms are typically equipped with two-way audio and video and connect at least 
two sites; the main site at which the instructor is housed and one remote site (Reed & Woodruff, 1995). Some of the 
universities that use this technology own the connected sites; other institutions form a consortium consisting of sites 
owned by a number of universities or colleges. 

Many educational institutions have adopted the technology. Interactive television-based classrooms clearly 
have been at the maturity stage of the product life cycle for some time. The vast explosion of the World Wide Web 
and Internet technologies have led to an increase in Web-based course offerings and, therefore, driven ITV 
classrooms into the maturity, perhaps even into the declining, stage of the product life cycle. 

Background 

The institution is a comprehensive university located in Minnesota and has approximately 15,000 students. 
The university has been involved in ITV for approximately twenty years and is a member of a Distance Learning 
Network. The network consists of two separate networks and has six members and numerous sites. Members are 
responsible for providing technical support to users and for the maintenance of the classrooms. 

Several of the ITV classrooms at this university are less than three years old and are equipped with video 
cameras, a smartboard, and one computer at the instructor station. The instructor carries a microphone, and the room 
is equipped with microphones in the ceiling. The remote ITV classrooms are owned by colleges or universities 
within the Minnesota State Colleges and University system and also have video cameras and microphones. This 
technology allows the transmission of real-time compressed audio and video. Some of these classrooms are 
equipped with a computer; other classrooms are not. 

The cameras in all ITV classrooms are voice activated and allow for interaction between sites. However, 
the instructor is only able to view one of the remote sites at a time if more than one remote site is connected with the 
main campus. On the other hand, audio is transmitted from all remote sites. Instructors control the volume or may 
mute the microphone; students at the remote site may mute their microphones. The instructor has the capability of 
switching between two cameras at the main campus: one camera either displays the instructor or desktop 
applications in use; the second camera shows the students facing the instructor station. 

Many students find an ITV course convenient because it eliminates the need for commuting to a distant 
campus. Individuals who reside in rural areas might not have access to the courses if they were not offered via ITV. 
Also, there are several disadvantages to this delivery system. One disadvantage is the limited instructor-to-students 
and student-to-student interaction deemed so important in distance education (Moore & Kearsley, 1995). Another 
drawback is the dependency on technology, which is the case in any technology supported distance education 
setting. 

Experiences in the ITV Classroom 

The instructor who taught in the ITV classroom of this university during Spring Semester 2003 had 
previously taught in this educational setting. He had taught the course many times and used well established course 
materials such as a syllabus, activities, and assignments. The course taught was a requirement for a school media 
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licensure, and four remote sites were connected to the campus where the instructor was located. Fifteen students 
were enrolled in the course. One of the sites had only one student present. 

Instructor 

While teaching this ITV course, the instructor experienced many frustrations associated with the 
technology. Numerous times, he entered the classroom and found not all sites were connected. During the first class 
session, one site did not come online until one hour after the session had begun. The instructor and students at the 
main site had difficulties in hearing comments of students at remote locations on a continuing basis. Students at the 
remote site did not fare much better. The instructor arrived approximately 15 minutes before class started in order to 
ensure that the technology was working properly. He never knew what to expect when he entered the classroom. 
During most class sessions, the technology did not work as expected. In fact, according to him, the technology got in 
the way of teaching the course. Negative experiences continued throughout the semester for the instructor and for 
the students. While these may have been isolated problems not typical of most instructors’ ITV experiences at this 
university, the expectation is that these issues should not occur with a product that has entered the mature stage of 
the product life cycle. 

Students 

At the end of the semester, students completed a course evaluations questionnaire. Items on the 
questionnaire related to the technology, the instructor, and the overall ITV experience. The majority of respondents 
(82%) indicated that they strongly disagreed or disagreed that the ITV system was always properly working at the 
beginning of class. Twelve percent of students did not think the classroom had used was comfortable and well 
equipped. However, at least 92% agreed or strongly agreed with positive stated items used to rate the quality of the 
instructor. 

Participants also were asked to offer suggestions and provide other relevant comments in the form of open- 
ended questions. When asked to describe their ITV experience, four students indicated that they were able to take 
the ITV course because of convenience. Three students complained about the poor sound quality and mentioned the 
audio and video were not synchronized. Other comments included that ITV is not the best learning environment, too 
many remote sites were connected, daily troubleshooting was disappointing, and that the technology used in a 
technology-based course was not working. The most frustrating of the elements the class reported was clearly the 
technology — technology problems and audio difficulties. One student wrote, “I was greatly disappointed in having 
to turn on and troubleshoot ITV every time I came to class .... But it is still better than driving to the university’s 
campus.” Another student wrote, “I found it to be convenient in that I did not have to drive, but something does get 
lost in the remoteness of it all.” However, students also commented they appreciated having ITV courses as an 
option, they would take another ITV course, and that it was a good experience overall. Recommended improvements 
by respondents were; decreasing the number of connected sites, using better equipment, improving the sound system 
and technical support, and rotating the instructor between sites. 

Product Life Cycles and Organizations 

Based on the instructor and student experiences, the authors wanted to know how this experience might fit 
into a product life cycle. The four stages of the product life cycle are: (I) introduction, (2) growth, (3) maturity, and 
(4) decline. When a technology product enters the maturity stage of the product life cycle, the product has moved 
from cutting edge technology to an established technology tool. At this stage, products usually have been improved 
through the addition of new features (Peter & Donnelly, 1998). The investment of resources is decreased because the 
majority of resources were allocated during the introduction or growth stage. One can assume a product in the 
maturity stage is consistent, predictable, and reliable. In case of ITV classrooms, students and instructors should 
expect to receive or deliver a good product, respectively. 

Not only do technologies go through a life cycle, organizations go through life cycles as well. As 
technologies and organizations reach the mature stage, one expects predictability, efficiencies, and customer 
satisfaction. Business owners have the same goal, because the profitability of a business is the highest at the 
maturity stage. 

In business settings, the goal is to reduce the time a product takes to reach the maturity stage. Franchising 
offers a model for trying to establish a mature business in a short and predictable manner. Here, franchise owners 
use standardized processes in order to deliver the same quality product at individual locations. They make extensive 
use of efficiency, economies of scale, and standardization, and promise the consumer the matured experience they 
expect. 
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Rationale for the Application of Business Models 

Whenever consumers purchase a product, they have certain expectations pertaining to quality, efficiency, 
and standards. In an educational setting, students have the same expectations. When instructors use new, emerging 
technologies, difficulties and occasional glitches are expected. However, when they use technology that has been 
well established and coined reliable — systems or tools in the maturity stage of the product life cycle — they have 
expectations that the delivery system will be functional the majority of the time. 

Business models focus on expectations of clients, standardization of process, performance, and efficiency 
of product delivery. Perceived quality is one major component in branding strategy (Peter & Donnelly, 1998). 
Without the application of a business model, every good experience pertaining to a product is a result of random 
events and, therefore, cannot be reliably duplicated. Is it possible that we can learn from the business world and 
apply these models in educational institutions? Can we gain from examining these business practices and integrate 
them? The franchise model is particularly interesting and could be applied in the adoption of technologies and 
development of support materials and services. 

Many books address issues businesses face at various stages. Gerber (1995) examines how entrepreneurs 
successfully start and grow small businesses, and addresses lessons learned from franchising. We could argue that 
faculty who develop new courses or experiment with integrating new technologies into their classrooms could be 
thought of as entrepreneurs even thought they do not face the large risks true entrepreneurs face. However, their goal 
should be to move their product (teaching) to a mature, dependable, and efficient model. 

The advantage of following a business model in education is that the institution gains efficiencies in the 
mature technologies and services so that instructors have time to explore and experiment with emerging 
technologies and services. For example, the instructor who taught in the ITV classroom during Spring 2003 devoted 
a lot of time to just surviving in the ITV classroom instead of exploring Internet capabilities. What should have been 
routine tasks required the same time and effort as would adopting and exploring new technologies. 
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Reinventing Education3 

Visioning and Developing a Model for Pre-service Education Using IBM’s 
Learning Village/Riverdeep Technology 

Marty Bray 

California State University 

Abstract 

The Carolinas Collaboration, a consortium of public schools and institutions of higher education in North, 
and South Carolina, was recently awarded a 1.5 million dollar Reinventing Education grant from IBM. Using IBM 
technologies such as Learning Village and River Deep the Carolinas Collaboration will implement standards-based 
reform, to improve teacher education programs, and to increase the effectiveness of the North Carolina and South 
Carolina statewide professional development initiatives. 

Introduction 

Since 1994, Reinventing Education has been the centerpiece of The International Business Machines (IBM) 
Corporation’s commitment to educational improvement, supporting school reform efforts and higher student 
achievement through the development of innovative technology solutions. The Reinventing Education initiative has 
provided significant grants to school districts throughout the United States to reinforce its mission to make a unique 
and meaningful difference in our schools. This year a consortium of public schools and Institutions of higher 
education in North and South Carolina received an IBM Reinventing Education 3 (RE3) Grant in the amount of 
$1,500,000. This consortium, known as the Carolinas Collaboration, consists of the Charlotte-Mecklenburg Public 
Schools in North Carolina, six school districts in South Carolina (York 1 Public Schools, York 2 Public Schools, 
York 3 Public Schools, York 4 Public Schools, Chester Public Schools, Lancaster Public Schools), as well as several 
institutions of higher education including the University of North Carolina at Charlotte, Davidson College, Johnson 
C. Smith University, Queens University, and Winthrop University. 

Improving Education Through Professional Development 

The Carolinas Collaboration seeks to implement standards-based reform, to improve teacher education 
programs, and to increase the effectiveness of the North Carolina and South Carolina statewide professional 
development initiatives. To this end the Carolinas Collaboration will use the implementation of technology as an 
integral component to improve the quality of teacher pre-service and in-service programs. This will be facilitated 
through the development of new and innovative teaching and learning methodologies via the innovative 
implementation of IBM Learning Village/River Deep technology. 

Learning Village and Riverdeep in the Classroom 

Learning Village was originally developed by IBM to provide educators with a web-based platform for 
instructional development. This year IBM has announced the development of a significant upgrade to Learning 
Village known as Riverdeep. This system utilizes on IBM’s Websphere technology. By moving to the Riverdeep 
platform educators will have a more robust system that conforms to the latest Internet content delivery standards. 

The vision for the New Teacher Preparation Program initiative is to develop the capacity of pre-service and 
classroom teachers to plan and implement effective instruction that will increase student achievement. The major 
challenges being addressed are that beginning classroom teachers are not adequately prepared to meet the demands 
of the first years of teaching and teacher retention. This problem centers on new teachers’ needs to better 
understand and apply effective instructional planning and best practices. The solution is ongoing, clear 
communication between teacher preparation programs and school districts regarding needs in instructional planning 
and articulation on the skills needed for effective teaching. To meet this need and fulfill this vision the Carolinas 
Collaboration will work together to align instructional planning and delivery across all teaching levels, including 
pre-service teacher education then use Learning Village/River Deep to help pre-service teachers develop 
instructional programs for implementation in the classroom. 

Using Learning Village/Riverdeep as a communications tool will strengthen teacher candidates’ capacity to 
develop lesson plans, utilize scoring rubrics, and utilize assessment tools. These are just some of the components of 
the Instructional Planner, the core element of Riverdeep. Riverdeep will also contain features that will facilitate the 
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assessment of pre-service teacher candidates as they progress through their course of study. Using Riverdeep, a 
model will be developed that will help Colleges of Education prepare pre-service teachers for the classroom while 
meeting the new NCATE Comprehensive Candidate Assessment Plan though the implementation of an electronic 
assessment plan. 

This presentation will highlight the model being developed by the Carolinas Collaboration and how that 
model leverages the new features of Riverdeep to help teacher candidates meet the challenges of today’s classroom. 



57 




Return On Investment in Nature Conservation — Evaluating Leadership 
Development in the World Wide Fund for Nature 

Jeroen Breman 

WWF College for Conservation Leadership 

Diederick Stoel 
Profitwise 
The Netherlands 

Background 

The World Wide Fund for Nature (WWF) is one of the world's most active and professional conservation 
organizations. Since 1961, WWF has been involved in programmes around the world to try and restore the balance 
between human activity and nature. Over those 40 years, WWF and other environmental groups can claim some 
outstanding successes. But, the bottom line is that while WWF may be winning some battles, they are certainly 
losing the war. The Living Planet Index (2002), which tracks macro-level global biodiversity, shows that the world's 
species and habitats have declined 35% over the past 30 years. The world will be biologically poorer tomorrow than 
it is today. The blunt truth is that we must do more. 

Conservationists face increasingly complex and large-scale challenges. They need to excel in technical areas, and 
also need to be able to engage and motivate partners: civil society, local and national governments, business and 
industry, the media and local communities. WWF needs to develop and motivate their teams and networks. In short, 
conservationists need a complex set of skills to carry out our jobs effectively. Above all WWF needs leadership. As 
one College participant put it, 'This is not just a biologist's job anymore'. 

The WWF College for Conservation Leadership was conceived in 1998 to help WWF and its partners meet these 
challenges and to develop the leadership skills needed to start winning the war. 

Programme Design 

The WWF College for Conservation Leadership aims to enable individuals to recognize, challenge, and 
develop leadership talent in themselves and others in order to improve the impact of nature conservation. The 
College strives to do this by delivering individualized online learning programmes, and creating learning 
environments to share challenges, knowledge, and approaches. 

The College is an interactive learning network. It combines new technology with traditional face-to-face 
training and offers a unique learning environment. Participants learn from each other, and analyze and improve their 
own performance to become better leaders and thereby achieve greater conservation results. The College promotes 
the ideas of self-directed learning, peer learning, learning on the job and continual improvement. 

The Conservation Leadership Programme is an 18-month programme with modules around four themes 
that have been identified as important for leaders in nature conservation: Leadership and Management, Strategy and 
Planning, Campaigning and Advocacy, and Communication and Networking. 

The programme is designed and developed in such a way that it can easily be part of the workday, with a 
time investment of an average of two to four hours a week, and modules that can directly influence the way that 
participants do their work. The College brings people together from around the WWF Network to share experiences, 
discuss new trends, and help develop the competencies that make up a leader in conservation. The College is not a 
just a place were participants take. They already have a lot of knowledge and skills which they bring to the College. 
The College is THE place to learn, share, and lead! 

Each year, through a competitive selection process that involves both the applicants and their supervisors, 
two groups of 25 participants, balanced in terms of geographic regions, gender and functional specialties, start the 
Conservation Leadership Programme. The programme begins with a one week workshop which sets the stage for the 
rest of the 18 months. Through role plays, exercises and discussions, this interactive workshop introduces the themes 
of leadership, communications, and information communications technology. Participants evaluate themselves on 
the key competencies within each of the four themes, and then in consultation with a facilitator, plan a customized 
study programme based on their needs. This week also provides an opportunity to get all participants familiar with 
the College’ online learning environment, the College Campus. 
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A primary task of the facilitators (both in the workshops as well as online), is to ensure a two-way flow of learning. 
Rather than having an expert stand in front of the group and lecture on the ‘right’ way to do things, participants are 
encouraged to apply what they know and share ideas with other participants to learn from each others’ experiences, 
add some fresh perspectives to their own point of view, and thereby increase the skills in their toolkit. This has the 
added benefit of building a greater capacity for learning and problem-solving by using their networks in the future. 
Participants are exposed to new theories and models that they are then asked to apply in their daily work situations. 
Following the opening workshop the programme continues with nine months of online distance learning using the 
College website. Participants select and take part in a series of six- week modules, requiring 2-4 hours of study per 
week. These modules are facilitated by a content expert, who reviews online assignments, answers questions and 
moderates discussions. Participants must satisfactorily complete all assignments to finish a module. There are two 
core modules. Leadership Skills and Effective Communications, that all participants have to complete. Next to the 
core modules, participants need to complete at least five elective modules that can be selected from a list of sixteen 
modules spread over the different themes. 

The group comes together again mid-way through the programme, for another week-long workshop. This is 
an important time to review progress, revise the study programme as needed, and address any problem areas. Also 
during this interactive workshop participants work on leadership, teambuilding, strategic partnerships and 
campaigning through role plays, exercises and a group project based on a real-life case study from the region of the 
workshop location. 

This second workshop is again followed by a further nine months of online learning. At the end of the 18 
months, successful candidates receive a Certificate of Conservation Leadership. 

The College started with a pilot phase of four groups. Even before Group 4 finished, WWE’s senior 
management decided to continue the programme. Since then, five more groups have started their 18-month 
programme. Of the first four groups with a total of 70 participants, 30 successfully completed the programme. Of the 
participants that started and finishes since the pilot phase, 75% successfully completed the programme. A handful 
others had valid reasons to ask for extension of their programme and may graduate in the next couple of months. 
Reasons for allowing extensions are amongst others: health problems, family crises, and module scheduling 
conflicts. 



Purpose of the Evaluation 

In an ideal world, evaluation is planned before or during the design of a programme. As in many cases, this 
was not done in the design of the Conservation Leadership Programme. 

Until now, reporting of successes in the College was very activity based. Eor instance locations of 
workshops, numbers of participants starting their programmes and how many complete the programme is reported. 
Some post-module and post-workshop satisfaction ratings are available, but little is known about the quality of the 
programme — or more important — of the participants’ learning. And are new knowledge, skills, and attitudes 
actually used? Eor the College management, the main questions are: a. are the right things being done? and b. are 
things done right? 

At the same time, the global trend of accountability of all kinds of processes is not surpassing non-profit 
organizations such as WWE. Senior management of the organization is starting to wonder what the impact is of 
sending participants to and funding a training programme. Can the impact of a leadership training on nature 
conservation be measured? 

This evaluation study is a first attempt in WWE’s Conservation Leadership Programme. The main 
questions in this evaluation study are: a. are participants learning the offered knowledge and skills? b. are 
participants applying them in their work? c. what is the impact on WWE ‘s critical organizational measures? and d. 
can the impact be translated in monetary return? 



Method 

In the following paragraphs the method of the evaluation study is describe. Eirst, the theoretical background 
is explained. Next, the translation is made to the current study: subjects and instruments are described. Einally, the 
mechanism for isolating the effects of the programme on the result is clarified. 

Measuring ROI of human resources, training, and performance improvement programmes is becoming 
common in businesses around the world. Increased emphasis on understanding inefficiencies in other sectors such as 
the health-care, non-profit, and public sector, but also building accountability into their processes causes these sector 
to adopt the ROI process (Phillips, 2001). 



59 




This study used a five-level evaluation process, based on Kirkpatrick’s four levels of evaluation (Kirkpatrick, 1994), 
and Jack Phillips’ return-on-investment (ROI) process (Phillips, 1997). This process enables the measurement of 
reaction and learning (Level 1 and 2) at the end of workshops and modules. It also enables the measurement of job 
application of knowledge and skills (Level 3) and organizational results affected by the programme (Level 4). In 
addition the process offers a tool to calculate the programme’s ROI (Level 5). Phillips (1997) emphasizes that the 
higher levels can not be seen and measured without paying attention to the lower levels. 

Evaluation according to Phillips (1997) is much like putting together a puzzle with five pieces: a. The 
evaluation framework, which defines the levels at which programmes are evaluated as described above, b. The ROI 
model. Figure 1 shows the ROI methodology model, depicting systematic steps to ensure consistent application of 
the methodology, c. Operating standards and guidelines (see Figure 2) that build credibility in the process and 
support a conservative approach to ROI evaluation, d. Practice provides support for the final piece: e. 
Implementation, bringing together the pieces of the puzzle. 



THE ROI PROCESS 



TM 



Calculating the Return on Investment 
of Performance Solutions 

- HR Solutions 

■ Training Solutions/E-Learning 

■ Major Projects 

■ Technology Solutions 

■ Organization Development Solutions 

■ Change Initiatives 



EVALUATION PLANNING 



DATA COLLLECTION 



DEVELOP/REVIEW 
OBJECTIVES 
OF SOLUTION 



DEVELOP 
EVALUATION 
PLANS AND 
BASELINE DATA 



COLLECT DATA 
DURING 
SOLUTION 
IMPLEMENTATION 



COLLECT DATA 
AFTER SOLUTION 
IMPLEMENTATION 



ISOLATE 
EFFECTS 
OF SOLUTION 



1. SATISFACTION/ 
REACTION 

2. LEARNING 



3. APPLICATION/ 
IMPLEMENTATION 

4. BUSINESS IMPACT 



DATA ANALYSIS 



CONVERT 
DATA TO 
MONETARY 
VALUE 



ROI Calculation • Example 


EVALUATION FRAMEWORK 


Net Program/Project Benefits 

ROI = 

Program/Project Costs 


Level 


Measurement Focus 


1. Reaction, Satisfaction 
& Planned Action 


Measures participant satisfaction with the 
program/project and captures planned action. 




2. Learning 


Measures changes in knowledge, skills, 
and attitudes. 


Costs ol program/project $230,000 
Benefits of program/project ftst year) $430,000 
$430,000 - $230,000 

ROI ’ 0.87 x 100 87% 

$230,000 


3. Application 

4. Business Impact 

5. Return on Investment 


Measures implementation and changes in 
behavior in the performance setting. 

Measures changes in business impact 
variables. 

Compares benefits to the costs. 
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Figure 1. The ROI process (reprinted with permission of Jack Phillips) 
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THE ROI PROCESS™ 



Figure 2. Ten guiding principles of the ROI process (reprinted with permission of Jack Phillips) 



A. comprehensive measurement and evaluation process 

that generates six types of measures: 

> Reaction, Satisfaction, and Planned Action 

> Learning 

>• Application and Implementation 

>• Business Impact 

> Return on Investment 

> Intangible Measures 

This balanced approach to measurement includes a 

technique to isolate the effect of the program or solution. 

Results-based Solutions 

> Performance solutions/projects are initiated, developed and 
delivered with the end in mind. 

> Participants understand their responsibility to obtain results 
with programs/solutions. 

> Support groups (management, supervisors, co-workers, 
etc.) help to achieve results from performance solutions. 

> A comprehensive measurement and evaluation system is in 
place for each program/project. 

y Variety of approaches utilized to measure contribution, 
representing a balanced viewpoint. 

y Follow-up evaluations (Application, Impact and ROI) are 
developed for targeted solutions/projects and results are 
reported to a variety of stakeholders. 
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Ten Guiding Principles 

1 . When a higher level evaluation is conducted, data 
must be collected at lower levels. 

2. When an evaluation is planned for a higher level, the 
previous level of evaluation does not have to be 
comprehensive. 

3. When collecting and analyzing data, use only the 
most credible sources. 

4. When analyzing data, select the most conservative 
alternative for calculations. 

5. At least one method must be used to isolate the 
effects of the solution/project. 

6. If no improvement data are available for a population 
or from a specific source, it is assumed that little or 
no improvement has occurred. 

7. Estimates of improvements should be adjusted for 

the potential error of the estimate. 

8. Extreme data items and unsupported claims should 

not be used in ROI calculations. 

9. Only the first year of benefits (annual) should be 

used in the ROI analysis of short term solutions. 

1 0. Costs of a solution, project, or program should be fully 
loaded for ROI analysis 
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As mentioned before, the evaluation of the Conservation Leadership Programme was not planned when designing 
and developing the programme. Thus no assessment of knowledge and skills before participation was collected. Still 
the ROI process used in this study provides useful tools to measure learning, application and impact after the 
programme. And management of the College decided that this first effort is preferred over no effort to evaluate. 



Subjects 

All 44 participants from Group 4 and 5 of the Conservation Leadership Programme were included in the 
study. The participants, 29 male and 15 female, are spread over 27 countries in Asia, Africa, Central and South 
America, and Europe. They have a variety of functions in the organization, such as communications, finances, 
human resource management, and different functions in nature conservation units and projects. Group 4 graduated 
one year before receiving the questionnaire. For Group 5 this was three months. The groups were the most recent 
graduates of the programme. 



Instruments 

Reaction data are continuously collected after each workshop and each online module. In the case of a 
workshop on paper, after an online module with an online form. Some focus on learning of new skills is included in 
these forms, but questions about Level 2 were also included in the questionnaire that is described next. 

A questionnaire was developed to measure results at Levels 2 through 5. On a five-point scale, participants were 
asked to identify their degree of success in achieving the learning objectives for the different competencies 
underlying the programme (Level 2) and list the top five most frequently used competencies. For Level 3, 
application of skills, participants were asked to indicate on a five-point scale the perceived change in a list of skills 
that are used in the College to characterize the different competencies. This level also included a top-five of skills 
that impacted their work most. For impact (Level 4) a list of measures was developed in cooperation with the 
Steering Group of the College. This group — in the College called ‘Theme Owners’ is among others responsible for 
guarding the quality of the programme in the different themes of the College programme, and consists of senior 
management members from across the WWF-Network. Measures were combined in 8 categories: financial income, 
financial cost savings without loss of conservation impact. Communication and media. Campaigning, Networking 
and partnerships. Policy change. Conservation, Innovation and learning. Within these categories, indicators focus on 
hard data such as output, quality, costs, and time. 
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For ROI (Level 5), two types of data are necessary, programme costs and programme benefits. Costs for the 
participants include: salary costs for the time spent on the programme including benefits, tuition, and travel, 
accommodation, and meal costs for two workshops. For benefits, participant were asked to reflect on the indicators 
of the impact question and select three or more accomplishments that can be converted into monetary value. 

After two months and two reminders, a short survey was sent to non-respondents. In this survey they were asked to 
indicate what the main reason was why did not manage to return the questionnaire. 

Isolation method 

The most critical step in the evaluation process is one that is often overlooked: isolation of the effects of the 
programme (Phillips, 1997; Phillips, 2001). What part of the impact can be attributed to the intervention, in this case 
the Conservation Leadership Programme, and what other influences may underlie the results? Phillips (1997) and 
Phillips (2001) describe several techniques that have been used to address the issue. In the present study, participant 
estimates of programme impact have been used. 

For each benefit listed in the questionnaire, participants were asked to estimate the percentage of the 
improvement that was actually influenced by the application of skills learned in the Conservation Leadership 
Programme. Also, participants were asked with what level of confidence they made that estimate. For the calculation 
of ROI the most conservative estimate of the monetary value was used. 

For example, if a participant estimated a monetary benefit of $4,000 and 60% of the improvement is due to 
the Conservation Leadership Programme. The participant is 80% confident about that estimation. Then the most 
conservative estimation of the benefit will be 60% x 80% x $4,000 = $ 1 .920. 

Results 

Of the 44 questionnaires, 15 were returned. Only nine of the participants answered all questions. The other 
six answered the questions up until the questions on monetary values. Participants used comments like “hard to 
estimate”, “there is no reason to calculate this”, and “most of it is qualitative but still important” as arguments to 
stop their answers at Level 4. The difference in time between the end of the programme of the two groups and filling 
out the questionnaires did not show a significant difference in the results. Therefore the results are combined. 

In general participants react positive to this programme. One skill that is taught in the Conservation Leadership 
Programme is to be able to provide colleagues and staff with constructive feedback. Participants learn this skill 
during the first workshop and will use it in the evaluation forms. In Table 1, the average ratings of the main themes 
of the workshops are given for both groups. It is not possible to list all evaluation results of all modules. 



Table 1. Selected reaction data for College workshop main themes 



Kick-off Workshop 


Group 4 


Group 5 




Leadership 


3.72 


4.57 




Communication 


4.47 


4.75 




Vision 


4.11 


4.63 




ICT 


4.83 


4.21 


Follow-up Workshop 








Strategic partnerships 


4.32 


4.20 




Networking 


3.90 


4.24 




Campaigning 


4.55 


4.50 




Teambuilding 


4.00 


4.45 



Participants report valuable learning across the themes of the programme. The ten competencies with the highest 
degree of success to reach the learning objectives are listed in Table 2. The lowest average score on any of the items 
in this questions was 3.7. In Table 3 the competencies applied most on the job are listed. Motivating and directing 
individuals and themes, a competency within the Leadership and Management Theme of the programme was 
mentioned by eleven of the fifteen respondents. Vision and Strategic partnership each by ten participants. 

Table 2. Top 10 Level 2 data: Degree of success in reaching learning objectives 



Competency 


Score 


Capitalizing on knowledge sharing 


4.4 


Developing and communicating a vision 


4.3 
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Using communication to procure results effectively 


4.1 


Motivating and directing individuals and teams 


4.0 


Mobilizing the Network and other NGOs 


4.0 


Recognizing and using media opportunities 


4.0 


Managing a project through all its cycles 


3.9 


Building and managing strategic partnerships 


3.9 


Creating awareness and educating others 


3.9 


Using communication technology effectively 


3.9 



Table 3. Top 3 competencies applied most on the job. 
Motivating and directing individuals and teams 
Developing and communicating a vision 
Building and manage strategic partnerships 



When looking at the skills applied on the job (see Table 4), the skills learned in modules in the Leadership and 
Management Theme have improved most. The six most improved skills are found in this theme, before skills in 
Strategy and Planning, and Campaigning and Advocacy. In Table 5 the four skills that had the biggest impact on 
their job performance are listed. Here also leadership skills were reported most, followed by campaigning and 
advocacy skills. 



Table 4. Top 10 Level 3 data: Level of improvement of applied skills 



Theme 


Skills 


Score 


L&M 


Establishing clear directions and objectives 


4.1 


L&M 


Inspiring and motivating 


3.9 


L&M 


Finding common ground and getting cooperation 


3.9 


L&M 


Formulating and communicating a vision 


3.8 


L&M 


Empowering others 


3.8 


L&M 


Sharing wins and successes 


3.8 


S&P 


Maintaining a positive relationship with potential partners 


3.7 


C&A 


Presenting case clearly 


3.7 


L&M 


Analyzing strategically 


3.7 


L&M 


Communicating clearly 


3.7 



L&M: Leadership & Management; S&P: Strategy and Planning; C&A; Campaigning and Advocacy. 

Table 5. Top 4 skills having most impact on the Job. 

L&M Analyzing strategically 

L&M Formulating and communicating a vision 

C&A Using relevant communication based on target audience 

C&A Developing and implementing appropriate campaigns 



Table 6. Top Level 4 data: Degree of influence of training on organizational measures 



Category 


Measures 


Score 


I&L 


Knowledge and experience shared with team members 


4.2 


C&M 


Quality of message - style, format, packaging 


3.8 


C&M 


Reach - number of target audience exposed 


3.3 


I&L 


Time spent on training and development 


3.2 


C&M 


Continuity - distribution of messages over the year 


3.1 


N&P 


Dialogue with potential partners established 


3.1 


I&L 


Knowledge and experience shared on the web 


3.0 


N&P 


Strategic partnerships established 


2.9 


CON 


Magnification achieved 


2.9 
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CON 



Conservation Capacity 



2.9 



I&L; Innovation & Learning; C&M: Communication & Media; N&P: Networking & Partnerships; Con: 



Conservation 



Table 7. ROI Calculation 





Group 


Benefits 


Costs 


CBR 


ROI 


1 


4 


- 


15,676.32 


- 


- 


2 


4 


- 


16,898.40 


- 


- 


3 


5 


- 


13,826.00 


- 


- 


4 


4 


15,200 


6,312.80 


2.41 


141% 


5 


5 


38,321 


14,108.36 


2.72 


172% 


6 


5 


20,000 


15,330.00 


1.30 


30% 


7 


4 


- 


14,710.40 


- 


- 


8 


5 


- 


11,290.00 


- 


- 


9 


5 


67,500 


24,335.91 


2.77 


177% 


10 


5 


56,290 


15,663.70 


3.59 


259% 


11 


5 


- 


13,882.00 


- 


- 


12 


5 


4,955 


12,300.00 


0.40 


-60% 


13 


5 


14,155 


6,434.60 


2.20 


120% 


14 


4 


34,347 


6,940.00 


4.95 


395% 


15 


4 


8,000 


7,705.00 


1.04 


4% 


Total 




258,768 


195,413.00 


1.32 


32% 



CBR: cost-benefit ratio. 

For Level 4 data, the participants were asked to indicate to what extent they thought that application of knowledge 
and skills had a positive influence on organizational measures. Measures in eight categories were listed. Table 6 
shows the measure with the highest degree of influence. Results show that items in the Innovation and Learning, and 
the Communication and Media categories score best followed by indicators in Networking & Partnerships, and 
Conservation. 

In Table 7 the calculations for the ROI are given. For each of the fifteen participants the costs and benefits 
that they reported are listed. Next to the ROI the cost-benefit ratio (CBR) is given. CBR is a different way of 
calculating return. The CBR is calculated by dividing costs by benefits. ROI is costs minus benefits divided by 
benefits. For the six participants who did not report a benefit, a ROI could not be calculated. By default their ROI is 
- 100 %. 

The figures are a representation for the ROI of the individual participants. The figures in Table 7 do not 
include development costs and other costs the College has. For the individuals working in their individual offices 
and projects these are their real costs. When College overhead costs are not included, the ROI for this group of 
participants is 32%. 

For WWF as an organisation the programme costs should be included. The tuition paid by the participants 
does not cover all costs. The organisation invests in the developments of its workforce. When these costs are 
calculated per participant, the ROI for the fifteen respondents would be -29%, or for the whole group of forty four 
participants -76%. 



Table 8. Non-respondents survey 



Reason for not responding 


Frequency 


no time 


18 


other engagements 


7 


other priorities 


4 


the questions are too difficult to answer 


5 


the questions are irrelevant 


0 


I could not open the document 


1 


I didn't receive the questionnaire 


3 
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Twenty three out of twenty nine non-respondents, reacted on the survey sent two months after the questionnaire was 
distributed. More than one answer was possible. The main reason for not responding was time. Only a few 
participants thought the questionnaire was too difficult. 



Discussion and Conclusion 

The main questions in this study were if participants in the College are learning and applying the necessary 
knowledge and skills from the programme, what the impact of this learning on the organization is, and if ROI can be 
calculated for these impacts. 

Participants reported learning in all competencies of the Conservation Leadership Programme. On a five- 
point scale, participant rate their own learning between 3.7 and 4.4 on 13 different competencies. When comparing 
the competencies that participants list as being used most on the job (see Table 3) and the level of reaching the 
learning objectives of these competencies (see Table 2), it becomes clear that the most used competencies not 
always score the highest in the degree of success of learning. Developing a vision for conservation and reaching 
conservation targets through strategic partnership are currently emphasized as two of the main strategies for 
achieving WWF’s goals. The difference in scores in Table 2 are not very big. Further analysis is necessary to 
determine if the content of the programme should be adjusted to increase learning in these key areas. 

In the description of the background of the College, it was mentioned that “above all, WWF needs leaders.” 
The results of this study suggest that participants have improved their leadership and management skills in their 
daily work. Eight skills in the Top 10 Level 3 data in Table 4 are skills learned in modules from the Leadership and 
Management theme. The Conservation Leadership Programme appears to achieve its mission in developing 
leadership talent. 

The biggest impact of the College programme on achieving WWF’ s key organizational measures are 
knowledge sharing and quality of communication (see Table 6). In the Top 10 list, many learning and 
communication indicators score high. Conservation measures appear in 9-th and 10-th position in this list. The two 
categories listing financial measures (income, and cost savings) do not appear at all in the top, just like policy 
change, and campaigning. 

The fact that the participants were part of a knowledge sharing experience influenced this score. And in the 
College programme, communication is the topic of one of the mandatory online modules. This could explain the 
high score in these measures. Financial benefits and savings may not be so obvious for the participants. Applying for 
funds from either internal or external sources takes time. Also budgets are prepared once a year and there are limited 
possibilities for revision. This may cause that impact is hard to measure on the short term, but could still have an 
effect after a year. Still the College should further investigate if the programme can be improved to support the 
impact on the less influenced measures. 

The question if ROI can be calculated should be answered with yes. However it is not for everyone in the 
group of respondent an easy task. Participants who reported financial benefits from the programme can explain well 
what the basis is for their estimation. For example: new products developed, project proposals approved, new 
strategic partnerships, time savings, cancellation of projects, and effects of new training programmes implemented. 
When asked what percentage of these benefits are due to participating in the College programme, participants’ 
estimates vary from 30% to 100% with an average of 72%. They are very confident about their estimations. The 
level of confidence varies from 70 to 100% with an average of 88%. 

The nine participants reporting a return in financial terms report benefits that are sufficient for the whole 
group of 15 respondents. For every dollar that the fifteen spend on participating in the College 1 dollar and 32 cents 
was returned. 

The question remains what distinguishes the nine participants who did report a return in financial terms 
from those who failed to do so. With current economic uncertainties and an increasing emphasis on accountability, 
the ability to clearly indicate the impact of certain processes on achieving organizational targets could show to be an 
essential competency that every Conservation Leader should have. More research will be needed to identify the 
differences between the two groups. 
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Abstract 

The purpose of the Internet-based Systemic Change EPSS is to provide the systemic change facilitators 
with the tools, resources, and information they need in the process, and accessibility to the other facilitators and 
communities dealing with systemic change. Creating such a system requires a detailed analysis, comprehensive 
design and careful implementation stages. We analyzed the requirements for such an EPSS in a previous study. In 
this session, we present the development and implementation of this endeavor. 

Introduction 

The purpose of educational systemic change is creating a better educational system than the existing one. 
Systemic change requires collective effort from people in a specific change context, including community and 
school. However, these efforts would be void without guidance and planning. Jenlink, Reigeluth, Carr, and Nelson 
(1996) suggested a set of guidelines offered through a systemic change guidebook to help systemic change 
facilitators with their change efforts. 

An Electronic Performance Support System (EPSS) is a system that provides cognitive support to 
performers for complex information. Since the systemic change effort is a process including complex information, 
an EPSS is necessary to support the facilitators of systemic change. The motivation factors for designing an EPSS 
for the guidebook are increasing the effectiveness of the guidebook and creating a community of systemic change 
facilitators in educational systems change field. Eirst, the guidebook consists of 26 discrete events that are sequential 
during the change effort and 18 continuous events that could be at different stages of the change process. Each 
discrete and continuous event consists of several activities and sub-activities. Each of these activities makes about 
10 pages of detailed guidelines, considerations, and tools, which makes the guidebook so thick and heavy that it 
might not be practical to carry it to many meetings conducted during the systemic change effort. It would also be 
expensive to copy it for key participants’ use. Secondly, changing an educational system in a school district is 
complex and non-linear process. Obtaining guidance and help from other experts in the field may become crucial in 
certain stages of the change process. For these two important reasons, the authors of the guidebook decided to 
produce it in the form of an EPSS, using hypermedia and the Internet. A team of designers and developers started to 
work on the EPSS by considering the motivation factors for it. 



Definition and Characteristics of EPSS 



What is EPSS? 

Before explaining the developed EPSS project, it is helpful to understand the concept of EPSS and the 
situations in which EPSS is useful. This section provides information about current knowledge on EPSS in the 
literature and cases in which EPSS can be used as a performance support tool. 

An Electronic Performance Support System (EPSS) is a just-in time performance support tool that is used 
at the place of performance. The objective of an EPSS is “to provide whatever is necessary to generate performance 
at the moment of the need” (Gery, 1995). EPSS is also a complete system of performance support tools to 
accomplish required tasks in the job setting. After the introduction of the computer networks and the Internet, 
performance support systems became universally accessible and now designers have opportunity to develop 
Internet-based EPSSs. 

Historically, the roots of EPSS may reach to the mid 1940s. Due to rapid growing body of knowledge. 
Bush (1945) indicated the problems of storage, search, retrieval and use of information for the people who needed 
specific knowledge from immense amount of information. He explained an imaginary machine, which he called 
‘Memex’. According to Bush (1945), the Memex would allow people to conveniently access the specific 
information in large information systems; in this way it would support human beings by enlarging their memory 
capacity. By doing so, humans could do information retrieval, modification and manipulation faster and better than 
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with conventional methods. From this perspective, Memex may be considered as the ancestor of modem EPSS 
systems. 

The focus of EPSS is to help people perform rather than to learn. Cole, Eischer, and Saltzman (1997) 
described the term EPSS as providing ‘just-in-time’ knowledge while performance takes place. By this definition, 
learning may or may not occur because the main focus is to support performance, not learning (Rosenberg, 1995; 
Rossett, 1996). 

Since the main focus is performance, it is generally agreed that EPSSs have different attributes than instmctional 
systems in many aspects (Cagiltay, 2002). After reviewing the literature, Cagiltay (2002) asserted three main 
differences between EPSSs and instructional systems. First, the purpose of an instructional system is to contribute to 
the achievement of certain types of learning outcomes in learning process, whereas the primary purpose of EPSS is 
to help people perform during the time of the performance (Witt & Wager, 1994). Second, the utilization timing of 
EPSS and instructional systems are completely different. EPSS is utilized at the time of the performance when the 
need occurs, on the other hand, instructional systems take place before the performance (Cole et al. 1997). Einally, 
the structural difference between instructional systems and EPSS is another major distinguishing factor. While 
instructional systems are well-structured and their content is well-defined toward certain learning objectives, EPSS 
does not necessarily follow a predetermined sequential path to support the performance (Laffey, 1995). 

Brown (1996) lists four circumstances that are best for utilizing an EPSS: using immense amount of 
information to perform at competent levels, unavailability of experts, high expectations from the workers, and self- 
directed performers. Reeves (2003) provides a set of questions to determine whether an EPSS is necessary or not for 
performers in an organization. Reeves (2003) lists following questions to answer the larger question: "Is an EPSS an 
appropriate solution to this performance problem?" 

1 . Do performers have easy (better yet, constant) access to computing? 

2. Is the task/job that requires support complex enough to warrant an EPSS? 

3. How stable is the task/job? (If it changes often, an EPSS may be more appropriate than other approaches 
that are more difficult to maintain, change, and disseminate.) 

4. How critical is the task? What is the cost of non-compliance or poor performance? 

5. Is time available for support? (Some tasks are so time-critical that the notion of consulting an electronic 
performance support system is ludicrous.) 

6. Do performers possess the necessary characteristics to use an EPSS in terms of literacy, computer 
expertise, or motivation? 

7. Is turnover among potential users of the EPSS high? (High turnover may often justify an EPSS because an 
EPSS is often more readily accessible than many forms of training.) 

8. Are the logistics of getting people to other approaches (e.g., leader-led training) so complicated or 
expensive that an EPSS is a more efficient solution? 

9. Will the EPSS be used for empowerment of performers or to assure that they comply with specified 
standards of performance? 

10. Is the task frequently repeated? If not, an EPSS may be a sound strategy. 

1 1 . Are complex decisions involved in the tasks? 

12. Can an EPSS be supported/maintained? How will it be updated? 

Considering the context of educational systemic change, the EPSS utilization circumstances provided by 
Brown (1996), and the questions listed by Reeves (2003), an EPSS is an appropriate solution for systemic change 
facilitators to support them in the change process. 

Characteristics of an EPSS 

In the literature, EPSS is defined with different definitions. In parallel to that situation, several authors 
expressed different views about characteristics and components of EPSSs. As cited in Cagiltay (2002), first, 

Schwen, Goodrum, and Dorsey (1993) proposed four characteristics for EPSSs: information management, 
collaboration management, productivity through embedded guidance and work metaphor, and finally a problem 
solving environment that integrates basic tools, information management, collaboration, and productivity tools in a 
seamless environment. Secondly, Reigeluth (1999) describes four critical components of an EPSS: a database, an 
expert system, an instructional system and tools. Reigeluth (1999) defines an EPSS as a computer program that 
provides support for the performance of a task. According to him, an EPSS usually has four major components: a 
database, an expert system, an instructional system and tools. These components are explained below: 
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• A database. A storage area of information that is necessary for the experts to perform their tasks. A database 
should be equipped with appropriate menu interface and search functions for easy accessibility of information 
when it is needed. 

• An expert system. A system that helps experts decide on the performance steps or provides guidance to make 
decisions while the experts completing tasks. 

• An instructional system. A guidance or help system to show how the EPSS can be used best for the users. 

• Tools. Parts of an EPSS that help a performer to complete different tasks, such as collaboration, e-mail 
programs and Web browsers. 

According to Reigeluth (1999), a human computer interface system serves as an integrator of these components. The 
designed internet-based EPSS for systemic change facilitators nests all these components and combines them under 
a website umbrella. The following section discusses the functional specifications of the designed EPSS and explains 
each component in detail. 



Systemic Change EPSS 

Functional Specifications of the Systemic Change EPSS 

As discussed above section, an EPSS is an integrated system of performance support tools to achieve 
certain tasks in the place of performance. The goal of an EPSS is “to provide whatever is necessary to generate 
performance at the moment of the need” (Gery, 1995). With the wide availability of the Internet and computers, 
designers have opportunity to produce universally accessible EPSSs, which can be available at any time any place. 
EPSS for systemic change facilitators has been decided to be accessible via the Internet. The primary mission of the 
Internet-based Systemic Change EPSS is to provide the systemic change facilitators with the tools, resources, and 
information they need in the process, and accessibility to the other facilitators and communities dealing with 
systemic change efforts (Cakir, Tuzun, & Reigeluth, 2002). Considering the primary mission of the system, the final 
product provides more functions than merely an electronic version of the guidebook. 

Prior to the development effort of the EPSS, Cakir, Tuzun and Reigeluth (2002) conducted a detailed needs 
analysis by following a model developed by Tuzun and Cakir (2002). In this analysis stage, they analyzed the 
system from four different perspectives: 

Organizations’ or individuals’ needs (clients’ needs): What performance, how and in which direction performance 
will be increased. 

User analysis: Demographics and computer literacy level of the potential users. 

Scope analysis: Performed in two stages, content analysis and task analysis. Content analysis; what content will be 
covered in the EPSS. Task analysis; what tasks will be accomplished by using the EPSS. 

Technical analysis: Considering the technical aspects of the EPSS such as feasibility of technologies, connection 
speeds, Internet Service Provider (ISP) software support capability, browsers, and- site address. 

Analysis stage was conducted with the two systemic change facilitators engaged with the systemic change 
process in a school district located in Indianapolis, IN and two designers of the system. Based on the data collected 
in the analysis stage, functions of the EPSS can be categorized according to EPSS components explained by 
Reigeluth (1999): 



• The database. The database consists of several elements of the system. The main part of the 
database is the guidebook which is also main content of the developed system. Through taking the 
advantage of various interface technologies, appropriate chunking of the guidebook is necessary in 
the system. Other content parts of the database are frequently asked questions and resources 
related to the systemic change efforts. 

• An expert system. During the analysis stage, facilitators expressed a need for a guidance system 
that would provide key information and guidance about certain issues that are important during the 
events of the systemic change effort. 

• An instructional system. The analysis showed that the content and the required tools to accomplish 
the tasks needed a system that explained users how to use them. 

• Tools. Initial analysis showed that a tool that creates a community to exchange information 
between the facilitators is a necessity for the EPSS. Besides a community tool, a search and print 
the page tools are required for the system. 
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Development of the System 

The development stage of the systemic change EPSS is a synthesis of the emerged components in the 
analysis stage. These components were integrated in a human computer interface with using hypertext/hypermedia 
systems. Three main components are discussed below; interface, content and tools, and tutorial. The systemic 
change EPSS should be designed with considering the novice computer and Internet users. Dillon and Gabbard 
(1998) stated that for novice users, hypertext/hypermedia based systems should be highly structured to prevent the 
‘lost in hyperspace’ phenomenon to occur; hence the user interface should be simple, clear and understandable. 
Content and tools are considered together in the system and discussed in detail below. Lastly, in order to assure that 
users get the maximum benefit from the system a tutorial component is added. 

User Interface Related Components The interface component of the system must facilitate easy navigation and clear 
orientation for the user. Such a system is designed based on the frame feature of HTML. Using the frame feature 
enables designers to make changes on the interface independently from the content. The interface consists of two 
frames, a navigation frame on the left and a content frame on the right. While navigation frame contains the links in 
the EPSS, content frame displays the information that is directed from the navigation frame. 

1. Navigation Frame: The navigation frame takes place on the left side of the system’s interface. It is 
designed based on JavaScript from the http://www.treemenu.com site. One advantage of the tree menu is that the 
structure of tree menu navigation provides a clear sense of orientation for the users. By using the navigation’s 
structure, a user can reach any part of the system without losing the track of where s/he is in the site. The usability 
tests showed that the users had clear sense of orientation without thinking too much about where they are. Tree 
menu provides another advantage for updating the site, since the content and navigation frames are independent, any 
addition or removal from content or navigation frame can be implemented within very short time without rigorous 
experience on HTML. 

2. Content frame: The content frame is placed on the right side of the interface. This frame contains the 
title, search, print, contact and sitemap functions. The title indicates the name of the section in the site. Under the 
main title, functional tools, search and print page take place. Although the navigation of the system is clear and easy, 
sometimes it may be difficult to find specific information. By typing the keywords in search box, a user can reach 
required information. Each page has a print button, which generates a printer-friendly version of the page on the 
screen. Under the function buttons, the section title appears. Basically, the section title tells the users on which event 
they are. The content field shows the text from the information database of the system which is the guidebook, 
resources and other related information. Lastly, the system is hosted by a server at Indiana University and credit is 
given to the university at the bottom of each page. The footer bar also contains links regarding to privacy, copyright, 
site map and contact information. 

Content and Tools Related Components The Internet-based Systemic Change EPSS has 6 main components: 

1. Providing guidance to the facilitators: This component is the kernel of the system and is event based. It 
provides a facilitator with the big picture of currently shown event in the system, which contains the important 
issues about the event. It gives a general view to the users about the activities that are going to take place, the place 
of the event in the whole process and the points that need particular attention. Additionally, the user is able to access 
to the other parts of the system from this component. 

2. Electronic version of the systemic change guidebook: The guidebook is designed in a hierarchical 
structure as, from general to specific, phases, events and activities. The book has 6 phases, each of which contains a 
number of events, and each event has variety number of activities. This component is an organized way of 
presenting the knowledge of the book conveniently and providing easy accessibility to the book. 

3. Resources: This component of the site contains the resources that develop the skills and knowledge of 
facilitators about educational systemic change process. These resources are categorized as readings, videos, case 
studies, sample documents and software tools. The site coordinator maintains these resources and updates them as 
required. Eacilitators currently do not have permission to upload any resources to this component. 

4. Community: The purpose of community component is to bring people involved in systemic change 
activities, especially facilitators, in one accessible place. The community component enables users of the system to 
share their experiences, concerns, questions, resources, and documents related to their systemic change efforts. The 
main feature of community component is online threaded discussion groups. The discussion software can be 
adjusted according to the topics’ and users’ needs. The technology behind the threaded discussion group software 
includes JavaScript, PHP (hypertext preprocessor language) and MySQL database. Also the system has a directory 
of people who are facilitating different systemic change projects in order to introduce the members to each other. 
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The component also provides school districts with a knowledge base to find a facilitator who has experience with 
similar contexts. 

5. Glossary: This component provides a brief summary of the process of systemic change and related key 
concepts and terms about it. It has two purposes, first it provides a common language for the terms and concepts 
used in the change process, and second, it provides information about a brief summary of the systemic change effort, 
which is an article written by Jenlink et al. (1996). 

6. Frequently asked questions (FAQs): This component provides archived answers to questions frequently 
asked. The answers are provided by experts in the field. The content is fed by discussions and e-mail messages. 

Tutorial Component In order to help people to use the system most effectively and efficiently in terms of time wise 
and learning curve, the system provides an introductory ‘how to use the system’ manual. This section serves for two 
purposes: first, it is a real user’s manual on how to use the system and second, it is one of the diffusion of innovation 
tactics. 

At first glance, it may seem that the use of the system is easy and one can figure out the ways to use it. It is 
true that the use of the system is easy and its interface is intuitive, however in order to use it at the most efficient 
level, one needs to know some key functions of it. 

Despite the fact that computers and the Internet are ubiquitous in our everyday life, some people may not 
feel comfortable to use them because they do not receive enough guidance on how to use those systems. The manual 
serves as an ice-breaker for this kind of people. 

Benefits of the System 

The Internet-based EPSS for systemic change facilitators provides more information and tools to its users 
than a static website. One benefit of the system is to provide guidelines for each event of the change process. The 
consultation component of the system contains all information related to the events that the facilitators need to 
perform to achieve certain stages of the change process. Another aspect is that the EPSS has community building 
tools, such as threaded discussion groups and a members’ directory. These tools add value to the EPSS and give 
power to the users to reach each other easily and to share their knowledge. Additionally, through gathering the 
knowledge from facilitators who use the website, it provides an invaluable knowledge base for further 
implementations of systemic change efforts in different school districts. 

Another benefit of the system is its low maintenance and update costs relative to printed or optic media 
since the system is using online technology to disseminate the information via the Internet. The technology 
powering the system allows the administrators to maintain and update the site without much software knowledge 
and programming skills. The infrastructure of the system uses DHTML, JavaScripting and basic HTML, thus it does 
not require downloading high-end browser plug-ins such as Elash player or Shockwave, which allows users to 
connect the system with even very low bandwidth connections technologies, such as Bluetooth or cellular phone. 

Conclusion 

An EPSS is a system that provides cognitive support to a performer at the need of complex and accurate 
information. Educational systemic change is a complex and difficult process which needs certain stages to be 
accurately implemented. The developed EPSS aims to provide knowledge, guidance and necessary tools for 
systemic change facilitators in the change process. Specifically, the system provides guidance on change process, an 
electronic version of the guidebook, community building tools, glossary and frequently asked questions. A tutorial 
about how to use the system is added to increase the effectiveness of the system. 

The system leverages the function of the systemic change guidebook. Because the delivery method of the 
system is the Internet, it can be reached anywhere in the world by any facilitator. Since most of the administrative 
components are accessible through the Internet, the system can be maintained at a low cost and accessible at anytime 
and anywhere. The EPSS helps with the accumulation of knowledge contributed by experts in the field. Because of 
all these contributions, we perceive the EPSS as a feasible solution to similar projects, which require use of massive 
knowledge and precise performance. 
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Assessing Technology Integration in a 
New Teacher Education Program 

Robert Campbell 
Okanagan University College 

The Education program at Okanagan University College (OUC) began in 1989 in partnership with the 
University of Victoria (UVIC), a larger institution with a long tradition of teacher education in the province of 
British Columbia. In this partnership arrangement students took the University of Victoria education program at the 
OUC campus in Kelowna and were taught by OUC Education faculty. Upon graduation students received their 
B.Ed. degrees from the University of Victoria. This partnership was to remain in place until OUC could begin 
offering its own independent degree. Eor this to occur OUC needed to design a completely new education program. 
Eaculty and administration worked together to complete this task and in September 2002 the new OUC Bachelor of 
Education K-12 program was unveiled and implemented. 

The new OUC Education program is designed to guide students toward the best ways to engage in 
professional practice in a pluralistic post-modern context. The program is based upon the following foundations: 
constructivism, de-centered practice, experience and knowledge as interpreted phenomena, collaborative approaches 
to teaching and learning, reflective practice, and an emphasis upon information technology. The latter foundation 
was emphasized so that graduates of the program would have greater experiences and advanced skills at 
technological use. As these new teachers entered the profession, they would, thus, be able to integrate technology 
more effectively into their own classrooms. A theoretical perspective that was key to the development of this aspect 
of the program was the acknowledgement of the non-neutral nature of technology. Information technology would 
not simply be “plugged-in” to this new program as a discrete course of study, or tagged onto the margins of the 
curriculum or timetable; instead, information technology would be regarded as a prominent feature of the 
curriculum, inherent in the overall design of the program. It was determined that information technology would be 
integrated, available, and accessible in as many parts of the program as was feasible including practicum seminars 
and field experiences. 

A formative assessment undertaken during the developmental stages of this new program determined the 
resources and facilities required to establish a strong program foundation in information technology. The 
assessment was based upon the essential conditions for the general preparation of teachers established as part of the 
National Educational Technology Standards. These essential conditions are: a shared vision, access, skilled 
educators, professional development, technical assistance, content standards and curriculum resources, student- 
centered teaching, assessment, community support and support policies (I.S.T.E. 2000, p. 6). It is recommended that 
a combination of these conditions be in place at the university, the college or school of education so that prospective 
teachers can create learning environments that are conducive to powerful uses of technology (I.S.T.E., 2000, p. 6). 

The OUC Education Program Department Chair served as the chief program designer and his field notes, 
which included observations as well as anecdotal reports from administration, faculty, and technical support staff 
served as an estimation of the level of essential conditions that would be present at Okanagan University College at 
the time the new program would be implemented. These data provided a description of the existing presence and 
strengths of each essential condition for general teacher preparation. Through this assessment it became apparent 
that a number of the essential conditions existed at the Okanagan University College prior to the development of the 
program; however, it was found that other conditions were not present at all, or if they were present, required much 
stronger support and resource allocation to ensure that the program would operate successfully. The findings 
enabled administration and faculty to address particular essential conditions and strengthen those that were deemed 
to be weak. It was realized that the implementation of information technology in this program required both 
increased resource allocation and increased professional development. These increases would ensure that the 
essential conditions would be in place as the new program was launched and would remain in place so that the new 
program could continue successfully. 

Each of the ten essential conditions is discussed below with reference to the field observations and the 
actions undertaken to address the conditions that required improvement. The discussion of the actions taken in 
response to the assessment is significant in that it describes the technological requirements; support services; 
professional development; instructional strategies; and administrative responsibilities, initiatives, and directives 
necessary to enhance student teachers’ technological literacy in a new teacher education program. The results of the 
assessment are pertinent to those involved in program/curriculum design, instructional design, educational 
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technology and will also be of interest to teachers, teacher educators, technology support staff and program 
administrators. 

1. A Shared Vision The professional education administration and faculty share a vision for technology use to 
support new modes of teaching and learning (I.S.T.E., 2000, p. 6). 

Field Observations: 

Most instructors do not use technology in their teaching. The few that are using technology do not 
necessarily acknowledge its importance beyond that of a tool. When information technology is used it is 
primarily used to enhance traditional teaching methods. There is little evidence of technology transforming 
instruction into new modes of teaching and learning. 

The existing (UVIC) program is exclusively elementary with methods courses in nine different areas. 
Instructors generally follow and expect a strict adherence to discrete study of subjects and consequently little or 
no integration of these courses occurs. Several instructors regard time spent integrating technology as time 
away from their discipline. Most regard technology an external element to teaching and to their subject area 
specialization. 

A very conservative atmosphere exists in the program. A portion of the faculty want to remain in the 
partnership association, i.e. not change the program at all! 

Actions: 

OUC’s new program emphasizes instructional themes such as “Developing Instructional Strategies,” “The 
Culture of the Classroom,” and “Integrating, Assessing and Reporting.” These themes are taught in modules by 
“teams” of instructors. Short seminars in the subject specializations are also included in each module. The 
emphasis upon instructional themes and teams rather than individuals teaching only their specialized discipline area 
as a single course has helped break faculty of out a discrete approach to subjects and has moved faculty as a whole 
toward a more integrative approach to learning. Still, the acknowledgement of information technology within 
instruction remains problematic, particularly when a faculty member sees funding spent on information technology 
and not spent on resources for their specific subject specialization. 

2. Access Access to current technologies, software and telecommunications networks is provided for teacher 
education faculty, classes, and field sites, including technology-enhanced classrooms that model environments 
for facilitating a variety of collaborative learning strategies (I.S.T.E., 2000, p. 6). 

Field Observations: 

A well-endowed Educational Technology Center exists; however, most faculty members have little to 
do with it since it is used for instruction in discrete courses (i.e. The Computer in the Classroom). Students use 
the Center extensively for Web browsing, word-processing and printing. 

All instructors have computers in their offices but they are old computers networked only for printing. 
Instructors have ports in their offices to access the Web, but their computers cannot run Web-browsing software. 
Some classrooms have Web access but instructors’ computers are not powerful enough to make use of these 
ports. 

Internal faculty communication is through paper, almost exclusively. 

Even though many students have home e-mail accounts, as do most instructors, computers are used very little 
for communication in field experiences. 

Actions: 

Over the year leading up to the new program implementation (2001), all faculty members 
received new laptop computers and computer projectors were installed in all classrooms to take 
advantage of the Web ports that existed in these spaces. Now, all internal faculty communications 
(announcements and minutes of meetings) are done with e-mail and attached documents. WebCT has been 
implemented for classroom administration and several instructors make extensive use of this teaching support 
tool by posting learning resources and learning objects online. The faculty is planning to implement a 
ubiquitous computing initiative this year (a wireless zone in the faculty building). Access to field sites remains 
problematic; however, several instructors are beginning to use WebCT for communication with 
practicum students and sponsor teachers. Additionally, the faculty is experimenting with a new video- 
conferencing system that may prove very useful in communicating with sponsor teachers and supervisors in our 



74 




college centres and in field sites. As instructors become more adept at using Web-based resources for teaching 
and become accustomed to electronic conferencing, the problem of access for faculty and students may be 
resolved, perhaps quite soon. 

3. Skilled Educators Teacher education faculty are skilled in using technology systems and software 
appropriate to their subject area specialty and model effective use as part of the coursework (I.S.T.E., 2000, p. 

6). 

Field Observations: 

Instructors generally possess basic computer skills: word processing, e-mail, and use grading packages, but 
generally do not use information technology as part of their classroom teaching. This is primarily due to a lack 
of facilities and resources rather than a lack of skills. 

Although provincial curricula recommend technological use in all subjects taught, this is not significantly 
modeled in classroom instruction. Again this may be due, primarily, to a lack of facilities and appropriate 
software. 

Actions: 

Over the last two years as the new program was being developed and implemented instructors have 
been provided with professional development workshops (in addition to new laptops). This scenario, in 
combination with the installation of computer projectors in all classrooms and the availability of WebCT, has 
permitted many instructors to use Web resources in class as part of their teaching. Because of these resource 
additions there has been a noticeable change in the use of information technology in classroom practice. 
Generally, instructors’ skills in using information technology are improving. 

4. Professional Development Personnel in teacher education and field experience sites are provided with 
ongoing professional development (I.S.T.E., 2000, p. 6). 

Field Observations: 

• OUC’s computer services department provides some professional development workshops, however, 
these generally are application type of workshops (how to use Microsoft Word, or PowerPoint). They 
are not specific to the use of software packages in teaching or in the manner in which technology, 
generally, can be used to transform teaching practice. 

• Many of the students enter the program with advanced skills in information technology, in some case 
more so than that of their instructors. 

• There does not appear to be much time for professional development opportunities. Instructors are very 
busy teaching and supervising student teachers. 



Actions: 

Funding levels have been increased to provide professional development workshops during the summer 
months for Education faculty. Most recently (July 2003), David Moursund, who is well known for the many 
editorials and articles he has written for Learning and Leading with Technology, came to OUC and presented 
several workshops on problem solving and project-based learning. These workshops were very successful and 
provided instructors with resources and models to use technology to transform their teaching practice. 

5. Technical Assistance Technical assistance for teacher education faculty and students is readily accessible 
and includes expertise in the use of technology resources for teaching and learning in PK-12 setting (I.S.T.E., 

2000 , p. 6 ) 

Field Observations: 

The program has on staff a lab facilitator who is available to solve computer and general media 
problems for both faculty and students. 

The computer services department provides the general maintenance of faculty and lab computers. 

They are responsive to faculty needs and trouble-shooting is done efficiently. 
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Actions: 



The OUC program is very fortunate to have on staff an education lab facilitator/technician who is 
available to solve both faculty and student problems. Although this individual did not begin the position with 
specialized knowledge in educational applications of technology, experience gained from years of trouble- 
shooting and problem solving resulted in a very skilled individual who makes an valuable contribution to the 
technology aspects of our program. The presence of this individual in our program exemplifies the benefits of 
having technical assistance and support from an educational specialist. 

One important addition to the University College staff was a campus technological coordinator who has 
a very strong background in pedagogy (he was an elementary school teacher for 13 years). He has spent a great 
deal of time with Education instructors teaching them to use PowerPoint, develop Web pages and use WebCT 
to enhance their classroom practice. Instructors have responded very positively, likely because this individual 
knows what is needed, and is keenly aware of what can be accomplished is a given amount of time. He is adept 
at providing people with the right amount of information; enough so that they can use information technology 
effectively, but not so much that they become overwhelmed. Although he was hired to work with all faculty 
areas at OUC his presence has greatly benefited Education faculty and students. He initiated a highly subscribed 
and very successful “Student Teacher Technology Leadership Program” that prepares education students to 
provide professional development in information technology for teachers in the local school districts. 

6. Content Standards and Curriculum Resources Technology-based curriculum resources that address subject 
matter content standards and support teaching, learning and productivity are available to teacher candidates 
(I.S.T.E., 2000, p. 6). 

Field Observations: 

Provincial curriculum guides (integrated resource packages) are available as PDE files on the Web, 
most instructors, however, require that students purchase paper versions. 

Little software is available that is curriculum specific. 

Instructors will order curriculum resources (Lab packs of CDs: tutorials, education games, and 
simulations) but often not use them. 

Actions: 

Good content specific software for teacher educators remains difficult to find. Education faculty 
members are increasingly relying on Web resources in the form of Webquests and instructional objects to serve 
as teaching resources. The professional development workshops in using WebCT and in applying learning 
objects to teaching have increased instructors’ interest. 

7. Student-Centered Teaching Teacher education faculty and professional teaching staff model student- 
centered approaches to instruction in education coursework and field experiences (I.S.T.E., 2000, p. 6). 

Field Observations: 

Instructors tend to teach the way they were taught — teacher-centered instruction is the prime teaching method. 
There appears to be resistance to student-centered teaching — from both instructors and students. 

Some instructors claim that the specific nature of their subject specialization requires a teacher-centered 
approach. 

Actions: 

Increased comfort with providing students with more “agency” over their learning may come with 
further professional development. Presently, teacher-centered instruction remains the primary mode of teaching. 
However, the new program has a four-credit capstone exercise, the Guided Reflective Inquiry Project (GRIP). 
This exercise permits students to undertake an inquiry of the own choosing and present their findings to faculty 
and peers in a public forum. This project-based learning assignment conforms to many of the objectives 
outlined by Moursund (1999). It is decidedly student-centered and may serve to model the benefits of this 
method of learning for both instructors and students. 

8. Assessment Teacher education faculty and professional teaching staff model the integration of teaching and 
assessment to measure the effectiveness of technology-supported teaching strategies (I.S.T.E., 2000, p. 6). 
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Field Observations: 

Technology-supported teaching strategies are not all that evident in the program. There is not much to evaluate. 

Actions: 

The new OUC program has established an educational advising council comprised of external 
faculty and community stakeholders to provide formative evaluation of our program as it is 
developed. We are planning an external summative evaluation of our program once the new 
design is completely implemented. The success of any new technology-supported teaching 
strategies will be accounted for in this assessment. In addition to these evaluation processes, our 
program must be assessed and approved by the provincial accrediting body, the British Columbia College of 
Teachers. 

9. Community Support Teacher preparation programs provide teacher candidates with opportunities to 
participate infield experiences at partner schools where technology integration is modeled (I.S.T.E., 2000, p. 6). 

Field Observations: 

The Education program has an official partnership with a local community elementary school. 

However, there is little time available to plan and implement truly innovative activities through this 
arrangement. 

The Education program and student teachers receive strong support from the local school districts for 
student teacher placements; however, the integrative use of information technology is not generally modeled in 
most local schools. 

Actions: 

Time seems to be the major impediment to integrating activities between OUC and local schools. The 
most recent and positive step in this direction has been the “Student Teacher Technology Leadership Program” 
initiated by OUC’s Educational Technology Coordinator. This program prepares education students to provide 
professional development in information technology for teachers in the local school districts. It provides OUC 
education students with opportunities to teach technology while at the same time provides professional 
upgrading to 

10. Support Policies Policies associated with accreditation, standards, budget allocations, and personnel 
decisions in teacher education programs and field experience sites support technology integration. Retention, 
tenure, promotion, and merit policies reward innovative uses of technology by faculty with their students 
(I.S.T.E., 2000, p. 6). 

Field Observations: 

There is little technology integration presently occurring. 



Actions: 

It is unlikely that tenure, promotion, and merit will be tied directly to instructors’ technological use. 
However, tenure, promotion, and merit are connected to grants and funding and these are available for 
technological applications. Consequently, there is a connection between tenure, promotion, and merit although 
not a direct connection and certainly not one that is an established policy. Ultimately, teacher educators can find 
funding sources and research avenues in the areas of technology. Participating in technological innovation can 
yield substantial benefits in funding and can, indirectly, result in tenure, promotion, and merit. 
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Triggering in-depth argumentation through scaffolding tools 



Milton N. Campos 
University of Montreal 

Abstract 

This paper explores how argumentation procedures could be triggered by Knowledge Forum scajfolding 
tools in the context of a community of practice of nurses in the domain of health. We present a brief historical 
context of the constitution of this community of practice, describe the participatory design process, explain how 
communication was scaffolded by instances of arguments through use of the tool, and how community interaction 
was managed to enable a knowledge-building process sustained by conversational practices organized around 
arguments. In addition, we describe the discourse analysis method used to study the scajfolding of the collaborative 
argumentation process. Results suggest that high level of intentional use of the scajfolding tool, which might be 
related to previous research identifying a higher order level of ill-defined problem-solving and collaborative 
learning, are characteristic of knowledge building processes (Campos & Laferriere, 2002; Campos, in press). 

Objective 

The research aims to verify that a community structures collaborative knowledge through the use of a 
scaffolding tool, leading to a higher level of ill-defined problem-solving. Results suggest that the scaffolding tool 
assists problem-solvers to communicate in a structured way by applying the appropriate steps necessary to sustain an 
idea and to convince others of its importance, relevance and / or plausibility. The value of integrating online 
conferencing, be in totally online or mixed-mode contexts, has been widely recognized but, to our knowledge, there 
are no studies focusing on knowledge building scaffolding tools. This research aims to provide quantitative and 
qualitative evidence of the pertinence, adequacy and ability of conferencing systems, in particular Knowledge 
Forum, for the triggering of in-depth thinking and higher order collaborative learning. Studies point to the fact that 
(1) community management is necessary but not sufficient to support knowledge building, and that sufficiency 
might be achieved by the use of proper tools (Campos, 1998). In addition, cognitive research on reading and writing 
processes has shown that (2) the availability of databases and the possibility to revisit and work on saved 
“memories” enables conference participants to keep track of networked conversations and to build communities of 
knowledge (Scardamalia and Bereiter, 1994). We will present how the nurses of a pan Canadian community 
working on problems of their practices used the Knowledge Forum scaffolding tool and how it supported the 
structuring of their progressive communication process. The scaffolding tool was customized to support 
argumentation by helping the facilitator (1) and by enhancing the nurses’ reading and writing processes (2). 

Theoretical Framework 

When people participate in electronic conferences, they have written conversations. Paradoxically, those 
conversations neither have the formal structure of essay writing nor the informal character of personal 
conversations. However, although scarce, the literature has highlighted the fact that a conversation “in writing” 
allows the participants to reflect more consistently about their ideas because of the editing process that is involved in 
active reading and writing (Bruer, 1994, Scardamalia & Bereiter, 1994). This reflexive way of participation can 
induce networked partners to engage in collaborative learning and knowledge building. How? Piaget explained the 
process of awareness as the engine of change that scaffold new paths of understanding (1974). There is a clear 
difference between being able to succeed when performing an action and to understand it. This finding followed 
research to verify the difference between succeeding and understanding in the context of physical actions (children’s 
playing games, for example). The analogy is applicable to studies concerning discourse: a person can succeed in 
identifying a problem and structure it through language but to understand the problem, to lay down an argument, to 
identify the premises and to reflect upon them, and to solve the problem by putting forward an acceptable solution to 
the argument, requires logical reasoning (hypotheses formulation). 

The distinction between succeeding and understanding, points to the difference between cognitive and 
meta-cognitive behaviour, largely investigated by cognitive science in general and cognitive psychology in 
particular. Meta-cognition is an awareness of our own cognitive processes, or a vigilant state in which we are able to 
be aware of the necessary steps to transform a concept, a notion or an idea. The change from previously held 
concepts, notions or ideas to new ones (or transformed ones) might follow the scientific method (drawing 
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conclusions from data and arguing for the validity of the concept) or “popular” or “naive” methods (drawing 
conclusions believed to be true by the individual for reasons that are psychological). Conceptual change is, thus, an 
intentional and reflective cognitive process leading to higher order learning (be dealing with scientific concepts or 
psychological notions and ideas) as opposed to lower order learning which is mainly automatic and most of times 
genetically determined. When the networked conversation process is collaborative, concepts, notions or ideas are 
changed or transformed in a collective exchange, as is the case of network-enabled asynchronous written discourse 
processes. 

The understanding that scaffolds are important in education has been discussed since Bruner (1985). 
However, we have no knowledge of research on enabling argumentation in collaboration through the use of 
technologically embedded scaffolds. I hypothesize that scaffolding enables meta-cognitive processes as explained 
above. Piaget’s model of knowledge proposes that logic expresses abstractly the fundamental operations of the brain 
and his psychology proposes that meta-cognition is a developmental process of operating logically. Thinking, thus, 
is governed by procedures that progress along time in an active process of conceptual assimilation and 
accommodation leading to adaptation to the social environment. This process shapes new paths of understanding 
through recursive comprehension and interpretation. Because electronic conferencing is distributed, meta-cognitive 
processes are both individual as social. Salomon (1993) argues that distributed cognition is “social” and “individual” 
because although cognitions are not distributed and procedural knowledge is totally individual as well as 
representations, the social nature of language requests both distributed and individual processes. According to 
Scardamalia (2002), knowledge building is a progressive process of production and improvement of ideas that are 
important for a given community. Scaffolding tools might be one of the ways to trigger and to improve meta- 
cognitive processes in electronic conferencing because they require logical reasoning or inferencing drawn from the 
contributions of others, a distributed context that is individual and social. By using scaffolds, participants of a 
knowledge building community can build on concepts, notions and ideas that others have structured and “tagged” 
before. 



Social and Technological Context and Research Method 

The OIIQ - Order of Nurses of Quebec, in partnership with hospitals of three Canadian provinces and other 
institutions (CEFRIO, CANARIE, and Fonds de T autoroute de T information), formed a French speaking 
community of nurses with expertise in heart care to address a perceived public concern concerning heart illnesses, 
the main cause of death among Canadians. I collaborated with the Order of Nurses of Quebec by supervising the 
facilitation and community building processes, applying a participatory design methodology in a research-action 
context. Participatory design is mainly understood as a process of developing software with the active collaboration 
of users. I applied this concept not develop any software but as an intervention method designed to build the 
community with all those interested in it: the Order of Nurses, the hospitals and health centers, the facilitator, and 
most of all, the nurses. It is particularly important to state that (1) the Order of Nurses was the institution who 
launched the idea and negotiated agreements with the hospitals without the intermediation of the researcher, (2) that 
the facilitator was recruited by the Order of Nurses in a process in which the researcher collaborated actively, and 
(3) that the nurses recruited by the facilitator voluntarily joined the community. The main goal of the actors of this 
process was to shape a networked environment in which the nurses could share their scientific and practical 
expertise on heart care in order to come up with recommendations able to be transferred to the public. The Order of 
Nurses of Quebec chose the conferencing system (Knowledge Forum) in active consultation with the researcher. 

We designed, the facilitator and I, and in consultation with the nurses and the scientific direction of the 
OIIQ, all necessary steps to build the community. Before launching the community, three different sub-groups of 
nurses were personally introduced to each other. The first subgroup meeting gathered all nurses living in the 
Montreal area (province of Quebec) and municipalities located within 2 hours from the city such as Ottawa, in the 
province of Ontario. The second subgroup meeting gathered all nurses living in and around Quebec City. The third 
subgroup meeting, with nurses living in distant parts of the province of Quebec and New Brunswick, organized them 
around a telephone conference designed to introduce them to each other and to train them in the use of Knowledge 
Forum. After this first phase, and to ensure that all nurses were introduced to each other, the facilitator and I 
designed the KF environment. Many “views” were opened (conferences or forums are called “views” in Knowledge 
Forum). The first conference called “Welcome” was designed to provide administrative information by the 
facilitator and to give access to the other conferences. The second conference, “Presentation”, was designed to allow 
the participants to present themselves with a short biography and photo. A third conference called “Tips” offered 
tips on how to navigate through KF discovered either by the facilitator or the nurses. The fourth conference, “At the 
heart of our exchanges”, was the launching conference. Its goal was to identify which problems related to the 
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nurses’ practices were most appealing for discussion and most relevant for their professional lives as related to 
public needs. 

In addition to features that are common to many conferencing systems (like key-wording, word search), 
Knowledge Forum has some others that make it unique such as (1) Editing (users can edit their messages even after 
they were posted), (2) Annotation (users can annotate messages by inserting “post-its”), (3) Quoting (users can 
quote texts written by colleagues by simply dragging them inside their messages), and (4) Rise-above (users are able 
to synthesize messages by packaging them within an upper folder). However, it is the (5) Scaffolding tool that was 
used in my research to test the hypothesis that structuring discourse was an important factor in the consistent 
knowledge building and collaborative learning identified in previous studies (Campos & Laferriere 2002, Campos, 
in press). The KF scaffolding tool allows users to insert tags within the text in order to categorize their own thinking. 




Figure 1 . This screen capture shows one message in which the text was structured with the help of scaffolds 
“Problematique”, “Questionnement”, and “Opinion” (between the yellow brackets). 

In order to enable argumentation procedures to be intentionally organized, I conceptualized scaffolds 
analogous to some instances of arguments that I have been applying in the coding of conference transcripts 
(Campos, 1998; 2000; Campos and Laferriere, 2002; Campos, in press). I inspired myself partially on Toulmin but 
did not follow his procedures because it would be unrealistic to apply a fine-grained analysis conceptualized to study 
dialogues and texts, for large databases as those found in electronic conferencing. We used some of his instances as 
scaffolds and created others: 

(1) Claim - introduction of a contextual situation that expresses concerns or difficulties concerning the 
practice or beliefs held by the writer, affirming something 

(2) Data: introduction of facts, statistics, scientific data, research results or other works that have an 
influence on the practice and would support the claim, or else psychological reasons for the standing of an idea.; 

(3) Envisaged solutions: engaging in a process of hypothesis formulation that might provide an answer to 
the claim put forward and consistent with the data 

(4) Questioning: formulation of interrogations 

(5) Opinions: offering judgments concerning claims, data, questioning or envisaged solutions presented in a 
previous message as a way of introducing a new turn of argumentation, unrelated to the other scaffolds. 
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My goal was to enable the participants to anticipate what their arguments would be, recognize their 
structures, and apply the instances of arguments intentionally and meta-cognitively within their collaborative 
conversation without affecting spontaneity. I recognize that those scaffolds might be insufficient or incoherent if 
used in some discourse contexts, but the idea was that of “keeping it simple” and providing the nurses a minimal 
structure for their collaborative exchanges. 



Results 

Data 

The entire database consists of 545 messages and approximately the same number of annotations. For the 
purpose of this study, we chose to study two conferences: one in which problems related to heart care practices were 
identified (with 122 messages), and another one in which nurses worked together to prepare deliverables to address 
the problems identified (with 141 messages). In the first conference nurses explored the difficulties of engaging 
patients in the prevention of heart failure by encouraging them to share the responsibility for their treatment, and by 
participating in the development of nursing strategies that could help their condition. In the second conference, the 
nurses prepared a teaching instrument to be handed to patients to help them control symptoms and signs of heart 
failure and therefore to enable auto-surveillance. The criteria to choose these two conferences were (1) the fact that 
nurses discussed what they considered to be important issues in heart care in the first conference, which was the 
start-up discussion, and (2) that the issues identified in the first conference were discussed more in depth in the 
second. It was in this second conference that a heart care kit was conceptualized and produced in order to be 
broadcast through the website of the Order of Nurses of Quebec to help the public to take charge of their own heart 
health. It is important to note that there is continuity between the first and the second conference. I limited the 
number of conferences to be studied (there were nine conferences) due to the extent and complexity of dealing with 
a large database. In addition, given that those two conferences were the core of the nurses’ activities, I considered 
that they were enough to identify the way that nurses engaged in the use of scaffolds. 

The unit of analysis 

I worked with two analytical unit levels: a technological one (the message) and a meaning one (the 
sentence). To study the sentences, two coders broke up the message texts, and then met to discuss minor differences 
found in order to achieve consistency. I adopted the sentence as the human cognitive unit of analysis because every 
meaning system reflects an action (which can be physical or discursive) that has a subjacent logic that emerges 
within the appropriate context. Piaget explains that action schemes are the products of assimilation processes in 
which previous procedures related to sequences of movements are applied to new situations while assimilation is the 
process in which new or old objects are incorporated to known schemes (1959). From the language viewpoint, an 
analogy applies: Judgments are acts that put concepts together or apply them to objects. Concepts are systematic 
unities in which extension (logical) defines the class and in which comprehension consists of properties or relations, 
a predicate being itself a relation. In other words, the meaning unit is not a concept but the judgment. In terms of 
discourse, the minimal unit reflecting a judgment (which contains, necessarily, a verb) is the sentence. A sentence is 
“a word, clause, or phrase or a group of clauses or phrases forming a syntactic unit which expresses an assertion, a 
question, a command, a wish, an exclamation, or the performance of an action that in writing usually begins with a 
capital letter and concludes with appropriate end punctuation”. We add to this definition that the presence of a verb 
is essential and that sentences express judgments that are not only conceptual (scientific knowledge) but also 
notional (popular knowledge). 

General Statistics 

(a) Messages with and without scaffolds 



Messages First Conference 



With scaffolds 
Without scaffolds 
Sub-total 



Messages Second Conference 



With scaffolds 
Without scaffolds 
Sub-total 
Total 



II Number 



90 

32 

122 



II Number 



95 

46 

141 

263 



Percent 



73.8 



26.2 



100 



Percent 



67.4 



32.6 



100 
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(b) Sentences within and outside scaffolds (all messages) 



Sentences First Conference I 


1 Number I 


1 Percentage I 


Within scaffolds 


706 


69.9 


Outside scaffolds 


304 


30.1 


Sub-total 


1010 


100 


Sentences Second Conference I 


1 Number I 


1 Percentage I 


Within scaffolds 


741 


43.5 


Outside scaffolds 


962 


56.5 


Sub-total 


1703 


100 


Total 


2713 





(c) Sentences within and outside scaffolds (only messages with scaffolds) 



Sentences First Conference I 


1 Number I 


1 Percent 


age 1 


Within scaffolds 


706 


92.0 


Outside scaffolds 


62 


8.0 


Sub-total 


768 


100 


Sentences Second Conference I 


1 Number I 


1 Percent 


age 1 


Within scaffolds 


741 


70.8 


Outside scaffolds 


305 


29.2 


Sub-total 


1406 


100 


Total 


2174 





(d) Occurrences of scaffolds 



Scaffold type 


1 Conference I 




First conference 


Second conference 


Number 


Percentage 


Number 


Percentage 


Claim 


30 


13.5 


20 


10,8 


Data 


51 


23.0 


73 


39,5 


Questioning 


60 


27.0 


33 


17,8 


Envisaged solutions 


54 


24.3 


28 


15,1 


Opinions 


27 


12.2 


31 


16,8 


Total 


222 


100 


185 


100 



(e) Stand alone occurrences and co-occurrences of scaffolds 



Messages with 


Conference I 




First conference 


Second conference 


Number 


Percentage 


Number 


Percentage 


1 scaffold 


37 


42.0 


63 


68.5 


2 scaffolds 


25 


28.4 


15 


16.3 


3 scaffolds 


16 


18.2 


12 


13.0 


4 scaffolds 


7 


8.0 


2 


2.2 


5 scaffolds 


3 


3.4 


0 


0 


Total 


88 


100 


92 


100 
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(f) Occurrence of pairs of scaffolds 





Claim 


Data 


Questioning 


Envisaged 

solutions 


Opinions 


Conf 1 


Conf 2 


Conf 1 


Conf 2 


Conf 1 


Conf 2 


Conf 1 


Conf 2 


Conf 1 


Conf 2 


Claim 






8 


5 


19 


5 


9 


10 


10 


7 


Data 










16 


5 


11 


5 


23 


8 


Questioning 














13 


6 


20 


7 


Envisaged 

solutions 


















10 


8 


Opinions 























Examples of scaffold use 

Message written by the facilitator: 

Claim The problem concerning the patient decision to take charge of his/her health is related to the non the 
patient becomes responsible for his/her own health. 

Envisaged solutions The solutions envisaged here refer to the importance of considering the needs of each 
patient and that of following his/her level of adaptation to the illness in order to enable the nurses to guide him/her in 
the process of taking charge of his/her health. 

Questioning What are the nursing strategies that should be specifically designed to follow the patients in 
their process of taking charge of their health in order to help them change behaviours damaging to their health? 

What are the models of behaviour change that could guide our discussion? 

Message written by one participating nurse: 

Claim With psychiatric patients there are contracts made to enable them to take charge of their health and 
made them responsible for the follow-up. We have one contract concerning aid, another one concerning what has to 
be done to act effectively. . . Because they engage in the process of taking charge, most contracts are respected. 

Envisaged solutions A contract with cardiac patients is a solution to be considered. 

Discussion 

What can be achieved by the use of scaffolds? In the examples presented above, it is possible to verify that 
a process of conceptual change takes place. The facilitator structures his/her message presenting a “claim” (it is 
needed that patients take charge of their health), “envisaged solutions” (reference to previous suggestions already 
made), and “questioning” the participants to propose solutions to the problem. One of the nurses, under the scaffold 
“claim” asserts that with psychiatric patients at his/her workplace, contracts are being used. The envisaged solution 
(a hypothesis) is that this procedure could be analogically 

Although the total number of messages of the first conference is 90, and of the second is 95, stand alone 
occurrences and co-occurences do not match those numbers. Collaborative conceptual change can be identified here 
because the hypothetical collaborative process of networked argumentation enables the nurse to jump to another 
path of understanding and to propose a solution to the ill-defined problem at stake. In this nursing networked 
community, conference participants “built on” the contribution of others using explicit or implicit conditionals (IF 
we apply a technique to psychiatric patients THEN this could be a solution for patients with heart problems). In 
addition, those structures, when under scaffolds, can be clearly identified, as the case of inferencing and implicit 
hypothesis formulation of the example above shows. The result of this exchange is that participants re-assess and 
reflect on knowledge, and re-build previously held concepts, notions or ideas. When collaborative conceptual 
change occurs, then collaborative learning is very likely to take place too. After assessing a process of conceptual 
change through the identification of the subjacent conditional operations that make people change previously held 
knowledge (re-equilibrating, thus, meanings that guide understanding), collaborative learning can also be assessed. 
However, collaborative learning can only be achieved if there is evidence in the sequence of exchanges that 
conceptual change was definitely incorporated in the renewed discourse, either by affirming it or by re-transforming 
it in the direction of renewed concepts, notions or ideas. However, a change in concepts, notions and ideas through 
networked argumentation that become more or less established (stable) in the discourse (thus, collaborative learning) 
is not evidence of knowledge building. The change has to be profound, i.e. the resulting knowledge must be unique 
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and a truly collective result of many asynchronously interconnected minds, something that an individual could not 
achieve alone. Although the occurrence of conceptual change, collaborative learning and knowledge building, which 
indeed happened, is not the focus of this study (verification is presented in previous studies: Campos & Laferriere, 
2002, Campos, in press), the role of the scaffolds is. The question here is to understand whether scaffolds had an 
impact in the way the nurses participated and collaborated. 

No proof can be presented that the scaffolding tool effectively triggered more profound discussions. One 
could argue that discussion could be profound in the absence of scaffolds, and only an experimental design applying 
rigorous methodologies such as think aloud protocols could offer definite evidence of a possible active role of 
scaffolds in triggering conceptual change, collaborative learning, and knowledge building. However, such a design 
would be inapplicable in a context of participatory design research, would certainly impeach spontaneity, and 
arguably freeze interactions. I participated in and followed the whole process of this community, which was active 
during six months. As explained above (section “Data”), the first conference studied here was one in which the 
nurses explored the problems of educating patients and helping them to take charge of their health. It was a kind of 
brainstorming process in which they explored dozens of issues related to this subject in the search of the most 
appropriate deliverable. I would label this conference as being in a reflective mode. The examples presented above 
were taken from the first conference. As a result of discussions done in this conference, the nurses decided to build a 
heart health kit, which was fully developed in the second conference. The heart kit produced by the nursing 
networked community (http://www.oiiq.org/santecoeur/) was published on the website of the Order of Nurses of 
Quebec (http://www.infirmiere.net/nouveau_infvir/contenu/sante_coeur/index.htm). I would label this conference 
as being in a production mode. In addition to have resulted in a concrete object of knowledge, shared through the 
web, the reading of the database messages strongly suggests that the scaffolds had a positive role in the way nurses 
structured their thoughts. The fact that they had to think about what they have to say and structure instances of their 
arguments within scaffolds shows, at least, that: (1) scaffolds were intentionally used to organize the content of the 
nurses contributions (judgments), and (2) that intentional processes of this kind were necessarily meta-cognitive 
because it was very unlikely for the nurses to organize collaboratively the content of communication this way while 
being naive and unaware of his/her own thinking processes at the same time. 

Quantitative results, although merely indicative, support these claims. First, around three quarters of all 
messages written in the first conference and a third in the second presented scaffold use (at least one scaffold), as 
shown in Table (a). The difference between the conferences is understandable because, in the first one, problems 
were being actively explored and solutions actively searched, while in the second the nurses were in a production 
mode, i.e. writing down the content of the tool kit. This context explains why sentences within and outside scaffolds 
(all messages) followed a similar trend in the first conference but not in the second, as presented in Table (b). 
However, if only messages with scaffolds are analyzed, then a much higher level of sentence inclusion within 
scaffolds is found: 92% in the first conference, 70.8% in the second conference (Table c). These numbers show a 
consistent scaffold use, suggesting that nurses intentionally structured their thoughts in a meta-cognitive way. It is 
certain, though, that the use of scaffolds had a different pattern in the second conference as Table (f) on patterns of 
scaffold combination shows. However, not much can be said of this pattern without further studies because the 
scaffold “opinion” is misleading and could be presented as a “claim”, “data” or an “envisaged solution”. In addition, 
it is interesting to note the mechanics of scaffold use. First of all, in the first conference the nurses used to combine 
scaffolds in the messages: if all co-occurrences are combined the result is 58%. This trend is different in the second 
conference: if all co-occurrences are combined the result is 31.5%. Conversely, inserting just one scaffold was more 
frequent in the first conference (42%) than in the second. In this conference, scaffolds that occurred alone and hence 
were not combined with other scaffolds, responded for 68.5% of the occurrences, as shown in Table (e). It is 
interesting to note that the scaffold “opinion” occurred more frequently when alone. In the first conference, it was 
responsible for 54.1% of stand alone scaffolds, and in the second, 20.6%. As mentioned above, different from the 
other scaffolds, which are really instances of argumentation (“claim”, “data”, “envisaged solutions” and 
“questions”), “opinions” were sometimes “claims”, sometimes “data” presentation, and sometimes “envisaged 
solutions”. It is plausible to think that in some circumstances, it was “easier” to structure thoughts within a scaffold 
such as “opinion”. However, the negotiation process of establishing meanings to the scaffolds to be used, it had been 
suggested that the scaffold “opinion” would be pertinent for answers in which nurses would comment on the 
contributions of colleagues. If analyzed this way, the use of this scaffold shows a higher level of responsiveness in 
the community, especially in the first conference (reflective mode). It is understandable that its use fell in the second 
because “opinions” were less pertinent in a production mode in which only validated data related to the effective 
production of scientific content based on practice could be incorporated to the health tool kit. 

Last, but not least, the numbers concerning the occurrence of scaffold types shown in Table (d) confirm the 
nature of the different modes related to each conference, and provide indicators of the meta-cognitive character of a 
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great deal of interactions within this community. The basic elements of an argument (not doing a fine-grain time- 
consuming analysis) are: “claim”, “data”, “envisaged solutions” (warrant) and “questioning”. A person affirms 
something (“claim”) and, to defend his/her point of view, needs to present “data” to support the claim. Claims can 
be in turn disputed through “questions” that lead to hypotheses that are formulated as “envisaged (possible) 
solutions”. This sequence is not mandatory and variety is the rule in networked informal conversation, which is 
discourse far from the juridical framework which inspired Toulmin (1958). “Opinions” entered as a recursive 
movement that could be confounded with any of the previous instances of argumentation. The numbers are eloquent. 
In the first conference, they are reasonably balanced. However, it must be noted that the pair “questioning / 
envisaged solutions” are responsible for 51.3% of all scaffolds indicating a very high level of inquiry and 
hypothetical reasoning, strongly suggesting the plausibility of the reflective mode. Although the pair does not co- 
occur necessarily, as Table (f) shows, each of its scaffolds operates significantly with the other scaffolds. In the 
second conference, this trend changes significantly. Reflective thinking suggested by the use of the scaffolds 
“questioning / envisaged solutions” is still high but the scaffold “data” is responsible alone for 39.5%, which is in 
line with a production mode in content (data) for the heart health kit is what is really at stake. 

Conclusion 

Results show that the nurses involved in the heart health networked community supported one another in 
the knowledge building process and used the scaffolds consistently to structure their arguments. Most conference 
messages show the use of at least one of the scaffolds created to help participants to structure their thoughts through 
instances of argumentation. This fact, combined with all other aspects explored in the discussion, suggest that the 
hypothesis that scaffolds representing instances of argumentation had a significant role in the collaborative 
knowledge-building process of the community of nurses: they made claims, presented data to support them, 
questioned, hypothesized and proposed concrete solutions to the problems discussed. Many other aspects are still to 
be studied concerning the use of Knowledge Forum scaffolding tools in this community and others. However, this 
brief and introductory study suggests that this tool, customized with argumentation instances discussed in 
collaboration in a participatory design process, is worth using, enhancing and ameliorating. 
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KC RE ACHE: A “Brick & Click” Model of Shared Student Services 



David Cassiday 
Cherie Kelly 

Kansas City Public Television 

Introduction 

The Kansas City Regional Access Consortium for Higher Education (KC REACHE) is an alliance between 
Kansas City Public Television (KCPT-19) and nine colleges and universities in the greater Kansas City area that 
have pooled their resources and expertise to offer distance learning options to area students. KC REACHE is a 
unique alliance in that it includes two-year colleges as well as four-year, public colleges as well as private, large as 
well as small, Kansas institutions as well as Missouri institutions. KCPT, which serves both Kansas and Missouri 
around the greater Kansas City metropolitan area, is the coordinating and fiscal agent for the consortium. The nine 
member colleges include Avila University, Johnson County Community College, Kansas City Kansas Community 
College, Metropolitan Community Colleges, Missouri Western State College, Northwest Missouri State University, 
Park University, University of Missouri-Kansas City, and Washburn University. 

KC REACHE began as telecourse buying consortium in 1997. KCPT’s original role was to license the 
telecourses. As it became clear that the future of distance learning was online, KC REACHE redefined itself more 
broadly as a distance learning consortium and formalized its structure in December 1998. KCPT acted as the 
convener in this process and was able to do so in part because we are not competing with the colleges for students. 
KC REACHE members have created a remarkable culture of openness and sharing that did not exist previously. As 
we progress through the still relatively new field of online learning, each college has shared both its successes and 
its failures openly and in an atmosphere of candor. Each committee really represents an ongoing professional 
development association for its members. What the members learn from the committees, in turn, results in 
institutional development for each member's college. 

The chart below shows the governance structure of KC REACHE: 




The KC REACHE Board, which meets annually, has delegated operating authority to the Steering Committee, 
which consists of whoever is the chief decision-maker about distance learning at that institution. This includes 
Deans of Academic Affairs, Directors of Distance Learning, and Directors of Continuing Education, as well as the 
Director of Adult Education at KCPT. Pour sub-committees report to the Steering Committee: the Student Services 
Committee, which consists primarily of each institution’s Dean of Student Services; the Library Committee consists 
of Directors and Assistant Library Directors; institutional IT representatives sit on the Instruction & Technology 
Committee; Directors of Marketing recommend marketing strategies for KC REACHE. The Steering Committee 
and each sub-committee meet 2-3 times per semester. The Steering Committee has charged each sub-committee to 
select one (or more) project per year to undertake and present to the Board at its annual meeting. 

Addressing the Needs of Distance Learners in the Greater Kansas City Area 

In 2000-2001, national research and our own internal research within KC REACHE pointed to the same 
problems: distance learners felt disconnected from faculty, other students, and the institution (Kendall, Moore, 
Smith, Oaks, 2001; Dirr, 1999). In addition, the dropout rate for online courses was approximately 20% higher for 
online courses than it was for on-ground courses. KC REACHE sought to address this problem in two ways; 1) by 
developing the “branch campus” model of shared services, which allows distance students to access selected 
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services from any member campus; 2) by developing communities of learners that connects online students with 
each other and with on-ground students. These services would go beyond what is known as the “administrative 
core” (WCET LAAP Project, 2000-2002). We would place special focus on adult, online learners. In the Fall of 
2001, KC REACHE received a three-year FIPSE grant from the U.S. Department of Education to develop these 
services. KC REACHE’s vision was thus validated on national level. 

Branch Campus Services 

Developing the “branch campus” services has generally gone smoothly. In the first two years of the grant, 
all KC REACHE colleges have opened up their libraries, their Testing Centers, and their Career Centers to one 
another’s students. Students indicate that they are enrolled at one of the KC REACHE colleges, and they are able to 
use these services at one of the other colleges, if that is more convenient for them. These services help enough 
students to show that they were worth developing, but no one’s resources have been strained. We expect that other 
consortia will be able to replicate this model fairly easily. 

Communities for Online Learners 

Bringing together online students in communities of learners has proved far more challenging. Our 
approach was based on the theory of “student involvement,” developed by Astin (1984), Tinto (1990), and others. 
Astin defined “involvement” thus: 

. . . the amount of physical and psychological energy that the student devotes to the academic 

experience. Thus, a highly involved student is one who, for example, devotes considerable energy 

to studying, spends much time on campus, participates actively in student organizations, and interacts 
frequently with faculty members and other students. 

For the purposes of the FIPSE grant, we were not concerned with the amount of time that adult online 
students spent studying, and by definition, these students did not spend much time on campus. What we attempted 
to increase was their interaction with students and faculty by opening up traditional student organizations to online 
students. We targeted academically related student organizations, especially if they were tied to an academic major. 
In this way, we believed we would increase their interaction with faculty and students in a meaningful way. This 
would lead to a greater sense of community and connection with the institution, thus leading to increased retention 
rates. 

While Astin’ s research was based on traditional age, campus-based students, we felt encouraged by recent 
research that indicated that adult students benefit as much as traditional students from involvement (Graham, Long 
Gisi, 2000). Our experiment was to test this theory on a subset of that population: adult, online students. 

Opening Up Existing Organizations 

The theory that online students would be interested in participating in organizations directly related to their 
field in interest encouraged us to develop relationships with established on-ground student clubs. KC REACHE 
established ties with four student organizations during the first semester of the grant (Fall 2001). These included 
chapters of SIFE (Students in Free Enterprise), MOSAIC (Making Opportunities For Students to Appreciate 
Inclusiveness), TOT (Teachers of Tomorrow) and Phi Theta Kappa (a Community College Honor Society). Two of 
these clubs (SIFE and TOT) focus on future careers. MOSAIC initiates activities around significant social issues, 
and Phi Theta Kappa provides social activities and a membership to put on your resume. Work with these groups 
included the provision of a web page on the KC REACHE website and monies for the groups to promote their 
willingness to work with online students, as well as partial funding for on-ground meetings and events that included 
online student participation. These provisions were made so long as the groups upheld their agreement to recruit 
online student members and arrange activities that would be of value to the distant learner. 

We learned several lessons from these pilot alliances with student organizations. Most importantly, we 
learned the importance of communicating to them repeatedly what KC REACHE is and what its mission is. The 
knowledge possessed by the high-level administrators who sit on KC REACHE committees does not necessarily 
filter down to the students. Since our organization and mission were unfamiliar to them, sustained effort did not 
always follow initial enthusiasm. 

At times, these organizations did not actively reach out to online students. When they did reach out, we 
often failed to create events that elicited online participation. In addition, all of these initial groups are based in 
community colleges, whose student organizations struggle tremendously to generate participation because all of 
their campus-based students commute. Reaching out to the online population, that doesn’t want to commute, added 
another level of challenge. 
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We at KCPT had anticipated this difficulty and attempted to use news groups as an Internet-based 
communication tool that organizations could use to reach online students. The newsgroups, however, are difficult to 
use unless one has above average technical savvy, and the web-interface that we created was unattractive and also 
somewhat difficult to use. We finally found an easy to use freeware bulletin board (Phorum) in the middle of Year 
Two. This has improved communication somewhat, but the challenge continues to be to create compelling reasons 
for students to visit it frequently. 

The inconsistent responses and energy levels of student-led organizations also proved challenging. It is 
critical that communication be conducted on a frequent basis when trying to coordinate meetings and events - if 
responses are delayed or inconsistent, windows of opportunity are missed, and momentum stalls. Some groups 
responded enthusiastically by providing web-page content that would reside on the well-trafficked KC REACHE 
website, but regular updates and the creation of interactive components proved too taxing to the student resources 
that were available. As mentioned, publicity was a prickly issue because we still did not have a consistent 
mechanism for communicating with students about time-sensitive events. 

Even when online students became aware of these organizations’ outreach, however, they frequently 
decided that they did not have the time to participate with a group they deemed to be essentially social in nature. 
Several of the organizations had participation requirements for their members that online students did not feel they 
could complete, either due to time constraints or the need to commute to a college campus. Much of this feedback 
came to us through our Year One survey of online students. It did not appear that these organizations were meeting 
the needs of online students. 

The next course of action, in Year Two, to integrate distance learning students and traditional on-campus 
students was built around a previously successful on-ground event in which students from several KC REACHE 
colleges had participated, although not collectively. A regular area event, the Heartland Pride Association’s Gay 
Pride Parade, had seen involvement by a number of KC REACHE schools’ students groups. KC REACHE set the 
goal of persuading LGBT groups from the KC REACHE campuses to work together to participate in the parade and 
to reach out LGBT online students. The Dean of Student Services at Johnson County Community College said that 
they had been fairly successful at providing community for LGBT students because of the anonymity that the online 
environment provides. The Student Services Coordinator for KC REACHE worked with Student Services 
representatives to contact campus diversity clubs. She encouraged these groups to organize and to walk in the parade 
as a KC REACHE group. KC REACHE agreed to cover all costs of parade participation. By this time, we had 
begun to develop e-postcards that could be delivered through our IT committee to faculty and students. KC 
REACHE used this method to communicate about the Gay Pride Parade to online students. KCPT provided 
students with an electronic bulletin for discussion and event planning. Promotion was also done through a KC 
REACHE booth at a local AIDS Walk event and work with the Lesbian-Gay Community Center of Greater Kansas 
City. Several student groups responded with enthusiasm, but when the parade was held, students chose not to 
participate with the KC REACHE group. 

Reasons for limited success with the consortium event were varied. Again, students were unfamiliar with 
KC REACHE and its mission, and unsure as to how it related to their mission. While LGBT organizations reach out 
to a marginalized population, they are nonetheless very traditional organizations in terms of how they function. The 
Heartland Pride Parade is somewhat carnivalesque in nature, and it may appeal more to traditional age students who 
need to affirm their LGBT identity. LGBT adults at a different developmental stage may not have found the event 
as appealing. We did see one KC REACHE member school marching under their own banner. 

Student Involvement vs. Professional Development 

What we discovered in no uncertain terms was that adult, online students do not want to get “involved” in 
anything that adds to their already hectic lives. Our surveys, our focus groups (coordinated by our EIPSE 
evaluators), and several failed attempts to create student organizations that intentionally reached out to online 
students confirmed this lesson. Adult, online students, who are typically holding down a full-time job and/or taking 
care of families are simply not interested in getting “involved” in more activities. They did indicate, however, a 
desire to participate in activities that advanced them toward their academic and career goals. They were interested 
in activities that they could put on their resumes. This realization led to a major conceptual shift in our approach. 

Instead of inviting adult students to engage in sustained involvement in organizations for which they would 
have some responsibility, we are developing a series of discrete, independent events that they can opt in and out of 
as they wish. A preference for periodic participation rather than sustained involvement is the key concept. Research 
has clearly documented that adult students have a consumer mentality when they choose colleges and programs, and 
part of this mentality is they expect to have no more demands on their time than are necessary (Roach, 2002). These 
events will include student feedback for online faculty, an online session with career counselors to discuss resume 
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building, interview skills, and job search tips. This model has the best chance of working with adult, online 
students. It is the model followed by our most successful online student organization. 

KOSO: KC REACH Online Student Organization 

One collaborative online student group that has seen success is KOSO (KC REACHE Online Student 
Organization), an organization comprised solely of online students from KC REACHE member schools. Begun as a 
student mentoring program, the group has evolved into an organization dedicated to providing a voice for online 
students. One goal of the original grant proposal to EIPSE was the provision of tutoring and mentoring for new 
online students by their peers (Aquino & Wilkinson, 2002). We discovered early on that there were too many 
liabilities involved with peer tutoring (e.g., time and expense of proper training, lack of participants representing 
each consortium institution, limited availability of student mentoring time and heavy demand for services). Peer 
mentoring by students who had been involved with online classes for several semesters seemed manageable. The 
pilot mentoring project involved ten KOSO students. Services were announced through the KC REACHE website. 
Mentors were excited about the project and anticipated questions from students about how online learning works, 
time management and requests for tips for non-traditional students. They were inundated instead with requests for 
technical assistance. 

The KOSO organization’s first mission change was to drop the mentoring program. As intended in the 
EIPSE grant, however, they would still serve as a student advisory board for KC REACHE. The group served as 
focus group participants for KC REACHE student services planning as well as a focus group for EIPSE grant 
evaluation. Energized by opportunities to have their collective voice heard by people in positions of academic 
power, they submitted proposals to present at various student services and online education conferences. Three of 
the KOSO’s student leaders led a panel discussion with approximately thirty faculty members at a regional distance 
learning conference in August 2003. They have also submitted a proposal to present at the NASPA student services 
conference in March 2004. These events showed us what adult online students are looking for. Meetings with 
KOSO members revealed that they are interested in professional development, networking, and other participatory 
events that will enhance their resumes. 

Communication with Students 

As mentioned previously, KC REACHE began with the very limited purpose of serving as a telecourse 
licensing consortium. When KC REACHE leaders decided in 1998 to broaden the mission to include 
comprehensive academic programs and student services, we faced the challenge of communicating that mission to 
staff, faculty, and students. Internal marketing has emerged as one of our biggest challenges. Many college faculty 
and staff do not promote KC REACHE to their students because they are unaware of it themselves. 

Early on in KC REACHE’s history, KCPT licensed WebCT and provided course hosting services for 
several of the colleges. Under the pressure to integrate courseware with new student information systems such as 
SCT Banner, Pipeline, etc., the course hosting function has migrated back to the colleges. KCPT ceased hosting 
courses at the end of Year One of the grant. Shortly thereafter, we settled on the e-postcard concept but no longer 
had control over where to place the announcements. In a recent meeting with KOSO students, we asked them to 
show us where a recent KC REACHE announcement appeared on their homepage. We found many inconsistencies 
in where the announcements are placed and how they are labeled. We also found that our instructions were often 
misunderstood with regard to what the announcement was supposed to look like. This is not due to lack of effort on 
the part of the colleges. It is the result of the information passing through so many hands before it finally reaches 
the student. 

We will continue to work on smoothing out this e-postcard process, but we also will attempt to work 
around it by using our capabilities as a public television station. While we will not be able to advertise time- 
sensitive events, we are creating TV spots that will drive viewers to our website, where they will find information 
about KC REACHE services and events. Students who are introduced to online learning via our advertising and 
website are perhaps more likely to pay attention to KC REACHE information when it is posted and more likely to 
be repeat visitors to the website. 



Evaluation 

Our EIPSE grant evaluators have proved invaluable for ascertaining what services students are interested in 
and how to most effectively deliver them. They created a survey that we distributed online through the IT sub- 
committee and attached to every online course at the end of Spring semesters for each year of the grant. Online 
students completed these surveys on a voluntary basis. The surveys asked students how likely they would be to use 
library, career, testing, and tutoring services, and how likely they would be to attend on-campus events. The surveys 
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included space for student comments. Students were given the opportunity to express why they felt what they did 
and what could be done to provide them with a more satisfying academic experience. 

The findings were revealing. Demographic information indicates that the majority of KC REACHE 
students, especially those who take courses almost exclusively online, are of non-traditional age and are involved in 
full-time careers and have family obligations. The vast majority are female. Narrative responses indicated that 
students are not interested in participating in on-going activities and clubs. More appealing are discrete events that 
allow students to participate as their time and interests permit. They are seeking professional development 
opportunities where they can learn specific skills, gain relevant information, work towards a specific cause, or, most 
importantly, participate in an activity that will benefit their resume. Resume-enhancement has emerged as a key 
attractor for this population. 

This, in turn, has led us to consider a different theoretical base for our final year activities. Schlossberg, 
Lynch, and Chickering (1989) confirmed that most adults return to college because some other major change has 
occurred in their life that creates a need for further schooling. Designing our activities, our website, and all of our 
communication with their theory adult life transitions may prove a sounder course than basing our activities on 
Astin’s involvement theory. Without explicit reference to Schlossberg’ s transition theory, our Student Services 
Committee had already begun to reconceptualize KC REACHE student services as services that would assist 
students at “transitional moments.” 

With transitions in mind, the Student Services sub-committee has been re-energized as they have 
considered new ways to serve for this population of students. The sub-committee is working on providing a 
modified “Eirst Year Experience” program for online students. This program would not be mandatory, as a 
freshman seminar is for so many new traditional students today. Instead, components of the program would be 
provided through one-hour online classes, KC REACHE events, web site information and resources provided on our 
KC REACHE CD ROM. Services would also include interactive components such as tests and measures to 
ascertain student learning styles and personality types (King and Hamilton, 2000). 

Conclusion 

When designing student services for adult online students for the final year of the EIPSE grant, KC 
REACHE will focus on developing separate, independent events that advance students in a clear way toward their 
academic and career goals. We will continue to develop KOSO as a vehicle to give online students a voice and an 
avenue for giving feedback about their online experiences within the consortium. Using Schlossberg’ s transition 
theory, rather than Astin’s involvement theory, as a touchstone, we believe that we will provide avenues for student 
community and development that can be replicated by other colleges and consortia. 
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Abstract 

In an information age, schools should integrate information literacy into school curricula and educate children 
to be information literate. The purpose of this study was to develop and evaluate an integrated information literacy 
curriculum in social studies for elementary level. The collaborative action research was used in this study. The 
researcher and a fifth-grade teacher collaboratively planned an integrated information literacy curriculum based on 
the Big6 approach. The methods of interview, observation, document analysis, panel discussion, and surveys were 
used for collecting data. The results showed that the information literacy could be integrated into related content 
areas, so that they both could be beneficial. However, it should be well planned in advance, in order to avoid the 
problems occurred in the process of integration. 



Introduction 

The overwhelming flood of information bombards us in an information age. A citizen of the 2T‘ century will 
encounter a continuous stream of news and information broadcasts, as well as the two-billion homepages and one- 
half-million images available through the World Wide Web (Kranich, 2000). However, more information will not 
create a more informed citizenry automatically unless people know how to use information efficiently, effectively 
and creatively. These digital illiterates usually do not know how to access suitable information competently, neither 
can they evaluate information critically, or distinguish the differences among various formats of information. The 
end result of this trend is that people in the information age cannot solve problems or make informed decisions in 
their real lives. Thus, many researchers suggest that schools should integrate information literacy into the context of 
classroom curriculum, and educate children to be information literates, who can recognize, locate, evaluate, use and 
create effectively the needed information (AASL & AECT, 1998; Breivik & Senn, 1998; Johnson, 1999; Todd, 

1996; Chen, 2001; Bruce, 1997; Pappas & Tepe, 2002). 

The social studies are the study of political, economic, cultural, and environmental aspects of societies in the 
past, present, and future (Barr, Barth, & Shermis, 1977; Czarra, Collins, & Smith, 1998). For elementary school 
children, social studies have several purposes. One of the purposes, which is congruent to the concept of 
information literacy, is that the social studies provide students with the skills for productive problem solving and 
decision making, as well as for assessing issues and making thoughtful value judgments (NCSS, 1989; Woolever & 
Scott, 1988). Furthermore, the National Council for Social Studies (NSCC) has been in the process of revising their 
standards for years and finally published a report called Curriculum Standards for the Social Studies (NCSS, 1994), 
which emphasized that all students need practice in information literacy skills. However, researchers warned that 
the process of information literacy required not only the learning of a constellation of skills, but also a new way of 
thinking in order to derive meaning from learning (Carey, 1997; Chen, 2001; Taylor & Patterson, 2000). 

Are there any strategies or models suitable for teaching a new way of thinking in social studies? The Big6 
approach is one of the possible strategies, which is a process model of information problem-solving that 
encompasses six unlocked stages. Despite several studies recommended this approach was appropriate for the 
integration of information literacy, little empirical research results have been verified yet (Fisenber &. Berkowitz, 
1999, 1996; Fdlund, 2001). In addition, this type of integrated information literacy curriculum has never been 
investigated thoroughly in Taiwan. Therefore, more research should be conducted to explore how to effectively 
integrate information literacy into social studies. Is the Big6 approach a good choice for the integration process? 

Are there any things need to be aware? And finally, what kinds of changes do school curricula have to make in 
order to put the integration of information literacy into practice? 

Research Questions 

The purpose of this study was to develop and evaluate an integrated information literacy curriculum in social 
studies for elementary level. Specific purposes related to the problems were as follows: 

1 . To develop an integrated information literacy curriculum in social studies. 

2. To investigate elementary children’s performance in the integrated information literacy curriculum in 
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the social studies. 

3. To examine problems occurring in the process of integration in elementary school curricula. 

Methods 

Research Design 

The collaborative action research was used as a framework in this study. The researcher and a fifth-grade 
elementary teacher (Ms. Chang) collaboratively planned an integrated information literacy curriculum and evaluated 
its effectiveness. This integrated curriculum was designed based on the Big6 approach. In other words, the set of 
six distinct stages that comprised the Big6 problem-solving method was highlighted. They were task definition 
(TD), information seeking strategies (ISS), location and access (L&A), use of information (UI), synthesis (S), and 
evaluation (E) (Eisenberg & Berkowitz, 1999). Within the study, the researcher and the teacher continuously kept 
on a dialogue and reflected on the process of the research. 

Research Site 

The research was conducted at Sun Elementary School (a pseudonym), which was in the urban southern part 
of Taiwan. The total of students in this school was about 1,700. This school had two computer labs which both 
could connect to the internet, and a library media center which owned about 8000 print and nonprint materials. 

The research site was a fifth-grade classroom. There were thirty eight students in the class. Within the 
students there were great differences in academic achievements and learning attitudes. As Ms. Chang, the 
collaborative teacher noted, “Their differences are so big. . .some are totally behind. Since they are just grouped into 
a class, so they are always disputing and arguing.” (int Chang, 11/23) Nearly all students were also busy with other 
after-school activities, e.g., mathematics, English, arts, etc. 

Ms. Chang has been an elementary teacher for eighteen years. Her personality was warm but assertive. She 
loved to utilize various learning resources to help children carry out different projects. 

Research Process 

1. Stage of Preparation (4/2001~9/2001) 

Eirst of all, the researcher collected a variety of resources about information literacy, Big6, and social studies, 
so that the direction of this research would be more comprehensible. Then the researcher searched for a 
collaborative teacher in the near area. As a result, Mr. Chang showed that she was interested in this research 
because she encountered some problems last semester when she taught social studies in a framework of inquiry 
learning (rft, 4/20). 

2. Stage of Collaborative Action (9/2001-12/2001) 

Since there would be a mayoral election in December, after several discussions, the researcher and Ms. Chang 
selected the unit of Government and People in the textbook as the social studies theme for integration of information 
literacy. This action project, called “Electing a Good Mayor”, provided students with a real social context, so that 
they could transfer the abstract concept into their real lives. The project was designed based on the Big6 for student 
exploration (Eisenberg & Berkowitz, 1999). Eirst of all, students were organized into six groups, then started to 
define the tasks involved in the project, then decided the best seeking strategies and located various types of 
information, e.g., books, internet, newspaper, or human resources. After getting all of the needed information, they 
read, viewed and heard different types of information. Next, the students synthesized the main concepts of the 
theme and presented their projects in suitable types of media. Einally, the teacher asked students to do a self- and 
peer-evaluation of the effectiveness and efficiency in all stages they have experienced. Within this action project, 
the different types of literacy like library, visuals, media, computer, and network were all emphasized and 
intertwined in the social studies theme. Periodically throughout these stages, Ms. Chang and the researcher met with 
each group students to check on their progress. 

Data Collection 

Research data collected included participant observations (vid), researcher’s field notes (rft), interviews (int), 
questionnaire (que), student notebooks (sn), and student written reports (sp). All the interviews were recorded and 
transcribed, and the observations were videotaped and transcribed too. These different types of data were chosen to 
best illustrate perception as well as to triangulate results for more credibility. 
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Data Analysis 

All of the data were organized and coded based on the types of data and date. For example, rft 4/20 meant the 
researcher’s field notes written on April 20. Then, these materials were read and analyzed continually by the 
researcher, so that salient categories could be identified. These categories thus represented different perspectives 
drawn from the whole data set. 



Results 

The Integrated Information Literacy in Social Studies Curriculum 

For the elementary level, there are three units in the social studies textbook each semester and each unit has 
to be completed averagely in thirteen periods. However, since the new curriculum put into practice in Taiwan last 
year, the instructional time was reduced. Thus, short of teaching time was always Ms. Chang’s concern, “Now our 
classes are in a tight time frame. We don’t have time to do any remedial instruction. Math, language art. . .are all 
behind schedule...” (int Chang, 10/30). For integrating information literacy into the curriculum in a limited time, 
more abstract and confusing concepts in this unit were discussed in the class, while fact-based knowledge was 
assigned to students for self-reading. The teaching strategies employed often were group discussion, group 
presentation and concept mapping. 

According to Ms. Chang’s observation, these fifth graders were not familiar with information skills such as 
library classification, use of encyclopedia, newspaper reading, as well as interviewing skills (int Chang, 10/10). 
Therefore, these topics were the main focuses in the information literacy of this integrated curriculum. Due to the 
space constraint, following table was the concise version of the integrated curriculum design (see Table 1); 

Table 1: The Integrated Information Literacy in Social Studies Curriculum (concise version) 



Learning Objectives of Social Studies 
1 . 1 Understand the process of government formation 


1.2 Understand the functions of government 


1.3 Identify the organization of local government 


1 .4 Identify the levels of local government 


1.5 Understand the ways of officials creation 


1 .6 Comprehend the meaning of election 


1.7 Solve social issues through a democratic process 




Learning Objectives of Information Literacy 






1 2.1 Take notes 




2.2 Understand library classification system 


2.3 Utilize various types of encyclopedia 


2.4 Read newspaper 


2.5 Define the task of the inquiry project 


2.6 Attain information seeking strategies 


2.7 Find needed information 


2.8 Use information 


2.9 Organize and present information 


2.10 Evaluate the results and process 


# 


Objec- 


Big6 


Instructional Contents 




Student Learning Activities 




tives 


Stages 


(in classes) 




(after classes) 


4 


1.1 


L&A 


• Teacher taught how to take notes and 


• Students answered the previewed 




2.1 


U1 


students took notes during the class.* 


questions using different information 




2.2 




• Teacher gave previewed questions. 


sources. 




2.3 




• Students role-played the process of 


• Students completed the library activity 








government formation and discussed 


sheet. 








IL. 

• Teacher explained the 


library 










classification system and the index of 










encyclopedia. 






3 


1.2 


L&A 


• Teacher taught how to 


read 


• Students answered the previewed 




1.3 


U1 


newspaper. 




questions using different information 




2.1 




• Teacher and students discussed 


sources. 




2.3 




government’s functions. 




• Students completed the newspaper 




2.4 




• Teacher explained the organization of 


activity sheet. 








local government. 












• Teacher gave previewed questions. 




4 


1.4 


TD 


• Teacher explained the levels of local 


• Students answered the previewed 




1.5 


L&A 


government. 




questions using different information 




2.1 


U1 


• Teacher explained the ways of official 


sources. 




2.3 




creation. 




• Each group completed the concept map. 




2.5 




• Teacher explained the Big6 


process 


• Students collected various sources of 
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2.6 

2.7 

2.8 




and gave the project. 

• Teacher taught how to organize 
information. 

• Students in groups discussed traits of a 
good mayor. 

• Teacher and students discussed a 
candidate’s concept map. 

• Teacher gave previewed questions. 


information about the selected mayoral 
candidates. 


2 


1.6 


TD 


• Teacher reviewed how to read 


• Each group discussed possible 




2.1 


ISS 


newspapers. 


information seeking strategies based on 




2.5 


L&A 


• Each group presented the candidate’s 


the concept map. 




2.6 


UI 


concept map. 






2.7 




• Teacher and students discussed 






2.8 




information seeking strategies. 




6 


1.6 


ISS 


• Teacher taught how to conduct an 


• Each group brainstormed a list of 




2.1 


L&A 


interview with a candidate. 


questions for interview. 




2.5 


UI 


• Each group interviewed the selected 


• Each group listened to the recorded 




2.6 




candidates. 


interview tape and classified the content 




2.7 






based on the concept map. 




2.8 






• Each group collected various sources of 










information about the selected mayoral 










candidates. 


6 


1.6 


ISS 


• Teacher clarified the contents of a 


• Each group synthesized all needed 




1.7 


L&A 


written report. 


information. 




2.1 


UI 


• Each group designed a poster in the art 


• Peer evaluated other group’s written 




2.8 


S 


class. 


reports. 




2.9 




• Each group presented its projects with 










a poster. 










• Peer evaluated other group’s 










presentations. 




2 


1.7 


L&A 


• Teacher arranged a mock election 


• Students reflected on the results and 




2.1 


UI 


activity. 


process of their work. 




2.10 


S 


• Students cast their votes. 










• Teacher and students examined 










problems occurred during the activity. 




1 


1.7 


E 


• Each group presented their reflections. 


• Students answered questionnaire. 




2.1 




• Teacher debriefed with the class about 






2.10 




the whole project. 





# meant numbers of classes. Each class lasted 40 minutes in Taiwan. * Students took notes in each class. 



Student’s Performance in the Integrated Curriculum 

On the whole, there were thirty-four fifth graders from thirty-eight stated in the questionnaire they liked 
this integrated curriculum. Some of their reasons were collected as the following: “I don’t need to dully memorize 
all the contents in a textbook” (que 25); “I felt I can learn things in the textbook with this teaching strategy. 
Meanwhile, I can learn many new things beyond a textbook.” (que 8); “I don’t have to just recite so many things. 
This teaching method let me experience the real situation and understand it truly.’ (que 7) 

As for the reasons for disliking this curriculum, the four students stated that: “I become anxious when I can’t 
find any needed information. I am afraid I will be detained in the school.” (que 18); “. . .because it is too much 
trouble, and I don’t know how to present my work.” (que 20) “We can’t cooperate well.” (que 38); “I don’t know 
how to arrange the found materials. They make me confused. . .” (que 28) 

As the comments above illustrated, most of the children had positive reactions to the integrated information 
literacy curriculum, while the students who felt negative to it were mostly lack of information skills. In fact, this 
was the focus of the integrated information literacy curriculum. The following were the analyses of student’s 
performance of the integrated curriculum in social studies and information literacy. Due to the limitation of the 
paper length, only four more important learning objectives were selected for detailed exposition. 
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• In the Area of Social Studies 

1. Comprehend the Meaning of Election 

The researcher led students in groups to discuss traits of a good mayor with the concept mapping strategy. 

The group four selected character, work experience, and education as the most important traits of a mayor in its 
presentation. They asserted; “. . .He needs to serve on the city council or has some experiences like those before he 
can be a good mayor. Such experience can make him be more responsive to the citizens. . .” (vid 4, 1 1/12) 

However, the issue of the government official experience became a dispute among students. One student 
noted, “I disagree with that. Serving on the city council doesn’t mean he will provide good service. He may get our 
money, and get nothing done.” The group four argued, “Nobody will vote a person who accepts bribes. The 
newspaper will tell it.” Another student delightedly concluded, “You see, being an official is not a necessity for 
being a mayor. His service records are the main issue.” (vid, 1 1/12) 

Afterward, with the company of Ms. Chang and parents, students in groups interviewed the mayor 
candidates, as another source to help students make unbiased judgment. Through the series of discussion, 
information collection and analysis, interview, as well as a simulated voting activity, the fifth graders’ progress in 
understanding the true meaning of election was shown in their notebooks: “Every citizen has the right to vote. We 
should appreciate it.” (sn25); ”We can’t elect him only because he is our friend. We have to know his background 
and policy.” (sn 14) 

2. Solve Social Issues Through a Democratic Process 

The unit of Government & People emphasized solving social issues through a democratic discussion. Thus, 
respecting the ideas of others was an important learning objective in this unit. However, fifth-grade students did not 
perform well in it within the whole process. The problems could be shown in the interviews with three group 
leaders: 

“When one has an idea, another says it’s not good. But he won’t say his idea. So we have to change idea 
from one to one; finally we have a squabble and run out of time.” (int ler, 12/21) 

“If we have 8 minutes for discussion, there are 7.5 minutes in a quarrel. Everyone has his own opinion, and 
don’t want to listen to others. I was beaten by my group members.” (int 4er, 12/21) 

“One guy wants this piece of material, and the other wants it too. So they start to fight for it and it is pulled 
apart. Then we lost our piece of information again.” (int 2er, 12/21) 

The problems of working on teams and lack of democratic literacy were also illustrated in students’ 
questionnaire, twenty-two from thirty-eight students felt teamwork was their most serious problem during the whole 
project. In other words, despite working on teams for a month, students still could not respect other’s ideas. In the 
last reflection activity of this project, many students felt sorry about the tense relationship with others. Ms. Chang 
thus promised to give them more chances to improve their democratic attitudes, (vid, 12/17) 

• In the Area of Information Literacy 

1. Attain Information Seeking Strategies 

Interview is one of the strategies for collecting candidate information. Ms. Chang facilitated students in 
groups to prepare higher-level interview questions and arranged the opportunity to practice them. As the following 
responses in the questionnaire illustrated, students felt it was the most valuable skill they have learned, “It boosts up 
my courage. I get some information not listed in the candidate’ flyers. It’s unforgettable experience.” (que 31); “I 
noticed there were many volunteers helping out. It means a candidate can’t win his election without other’s help.” 
(que 36) 

On the other hand, during the interviews some students said they had difficulty in using tape recorders and 
taking notes, “. . .1 am so nervous that I stammer a lot. I forget taking notes. The candidate doesn’t answer my 
question in detail, but I don’t know how to ask him more questions.” (int 3, 1 1/23); “Although I interviewed our 
teacher before, this time is different. It is a mayor candidate now. I forget recording. . .” (int 4, 1 1/23); “Ms. Chang 
should let us practice more interviewing tasks, e.g., making an appointment, so we won’t be afraid. . .” (int 25, 

11/23) 

As the comments above revealed, owing to the lack of chances to practice before, it was a great challenge to 
the students. Posing the following question, taking notes, and using the recorders were three tasks needed to be 
improved. These problems were also noted in the researcher’s field note, ’’They don’t jot down the notes right 
away. They don’t know how to ask questions if the interviewee doesn’t answer their questions. Most of them can 
sociably pose the prepared questions and have the eye-contact with the interviewee.” (rfn, 1 1/21) 

2. Synthesize and Present Information 

Synthesis, the fifth stage of Big6 approach, emphasizes the integration and presentation of information from a 
variety of sources to meet the information need as defined (Eisenberg & Berkowitz, 1999). Ms. Chang and 
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researcher asked each group students to present their mayoral candidate project to the full classes with a large poster 
and a written report. The criteria for evaluation included scope & depth, expression, layout, comments, and 
cooperation. 

Each group presented its project on the selected mayoral candidates, then provided their reviews. Other 
groups could question them, and they had to explain further. As the group six synthesized the candidate’s strengths 
and weaknesses in the presentation; “His strong point is that he loves to help people, while his weakness is that he 
likes to criticize a person.” (vid, 1 1/26) After that, other students asked many questions about this candidate, such as 
how did he criticize others? What has he done for helping people? Ms. Chang was very pleased with their 
presentations and the process of interrogation in the social studies class (rfn, 1 1/26). 



In the written report, the group two synthesized the candidate’s traits in a table: 



Category 


Eamily 


Policy 


Education 


Character 


Service 


Experience 


Suitable for a mayor or not 


Not 


yes 


So-so 


So-so 


yes 


yes 



Their remarks were that, “We find that this candidate has strong and weak points, but nobody is perfect. Based on 
the table, we find that this candidate has yes in three categories, while no in only one and so-so in two categories. 

So this candidate is suitable to be our mayor.” (sp 2) 

As the above data showed, fifth-grade students could synthesize and present information. They have 
mastered basically the Big6 stages. 

Problems occurring in the process of integration in elementary school curricula 

Time management and curriculum integration were real issues in this study. As shown in the table 1, the 
curriculum covered twenty-eight periods. Except the regular thirteen classes of social studies, some language arts 
and arts courses, as well as one Wednesday afternoon (for interviewing activity) were all drawn on for this project. 
Therefore, the lack of time has been great concern for the researcher and Ms. Chang. Ms. Chang was anxious, “I 
didn’t have time to cover the part of central government in this unit, and students didn’t quite understand legislators 
yet...” (int Chang, 12/20). 

Curriculum integration is the center of the curriculum revolution in Taiwan lately. However, during this 
study, the science, mathematics, computer and language art all have independent instructional units and different 
assessment methods. Ms. Chang stated, “Our kids have had many different assignments recently. The science 
teacher gave them a research project yesterday. . .” (int Chang 1 1/23) In addition. Sun Elementary School held 
various sports and music activities this semester. All of these separated activities made students unable to 
concentrate on one or two core projects. 



Discussion 

The results of this study indicated that through an integrated information literacy curriculum, based on Big6 
approach, fifth graders experienced a meaningful social action, and grasped abilities to access, organize, and 
evaluate information. In the area of social studies, they understood the meaning of election in a democratic society. 
However, performance in solving social issues through a democratic way was weak. Teamwork was the heart of the 
students’ problem. More team projects might be the solution to this problem, as Pappas and Tepe (2002) stated, 
since students needed time and many opportunities to develop their knowledge and skills. More research needed to 
be done on this issue. As for the area of information literacy, their performance in interviewing improved greatly. 
However, posing additional questions, taking notes, and using tape recorders were three tasks which should be 
improved. Eurthermore, most students performed well in synthesizing information. They could classify information 
according to the concept map, and provide their objective comments. Thus, in general, Big6 approach was suitable 
for integrating information literacy instruction across the contexts of subject areas. 

Lack of instructional time and curriculum not-integration were two serious problems in this study. In fact, the 
abilities of “utilization technology and information”, “active inquiry & research”, “independent thinking and 
problem-solving”, which were the centers of the new curriculum in Taiwan, all should be based on information 
literacy. However, information skills must be taught progressively and systematically. They could not be grasped 
and applied into different disciplines in a short term. These findings were consistent with those reported by 
Eisenbreg and Berkowitz (1999). On the whole, teachers, curriculum coordinators, and any persons who were 
responsible for curriculum development should do their best to integrate information literacy across all curricular 
areas. In this way, our children shall be successful in schools, in the workplace and in the world of the 2L‘ century. 

Educational Implications and Recommendations 

There were several important implications and recommendations for educational researchers and educators 
to consider from the results of this study. 
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1 . The information literacy could be integrated into related content areas of the curriculum, so that they both 
could be beneficial. 

2. The concept of information literacy should be clearly defined, included, and well planned in the new 
curricula, in order to solve the problems occurred in the process of curriculum integration. 

3. Teacher should arrange the whole semester or whole year curriculum which crossed different disciplines, then 
selected an appropriate theme for inquiring, in order to improve student’s ability to participate in social lives. 

4. The future study could investigate the teacher role and parent attitudes to the integrated information literacy 
curriculum, as well as the partnership between teachers and library media specialists. 
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Not Just Spinning Logos... You Can Develop Performance Supporting 
Web-Based Back-End Database Systems Too! 

Seung-Youn (Yonnie) Chyung, Ed.D 
Boise State University 

Abstract 

Web-based database systems are often utilized as performance improvement tools in various areas of 
organizations. A few examples include course registration systems, employee directory systems, customer 
relationship management systems, and library catalog systems. Recent web authoring/development software allows 
non-programmers to develop fairly sophisticated web-based database systems within a considerably short 
timeframe, at a low cost, and without having to acquire professional programming skills. In this article, I will 
describe an example of an in-house developed web-based student enrollment tracking database system, which was 
developed to facilitate the administrative workflow of a multi-institutional distance education program. I will also 
explain the main steps for developing the web-based database system, using Active Server Pages with specific SQL 
and VBScript code. 



When Spinning Logos Don’t Improve Human Performance 

Recent web technologies allow us to present multimedia elements such as animation and video clips in CD- 
ROM based e-learning materials or webpages. There are several animation software programs in the market, such as 
Flash and Shockwave. What are the main purposes of using animation in information presentation? Does it improve 
learning outcomes? Research seems to show inconsistent findings in terms of the benefits of using animation in 
learning materials. Some research studies have revealed that animation included in learning materials could improve 
learning outcomes (e.g.,Catrambone & Seay, 2002; Large & Beheshti, 1996; Norman & Patricia, 1999). Other 
research studies showed that animation did not facilitate better learning (e.g., Blankenship & Dansereau, 2000) and 
it rather distracted learners (e.g., Sperling, Seyedmonir, Aleksic, & Meadows, 2003). 

How about effects of using animation in webpages? These days, it is virtually impossible to navigate 
through webpages without seeing one or two spinning logos or animated icons inserted in them. What benefits do 
animated icons inserted in webpages give to viewers? Spinning logos may gain attention from viewers for a short 
period of time and improve retention of information they viewed (e.g., Rae & Brennan, 1998). There also seem to be 
several pieces of anecdotal advice on how animation can increase organizational performance such as sales (e.g.. 
Sharpies, 1999). But experts in web design warn that extensive use of animated elements in webpages can distract 
viewers and decrease the usability (Nielsen, 2000). More importantly, few animations inserted in webpages are 
designed as performance improvement tools that web viewers use to reduce their performance deficiencies or 
improve their performance levels. What many organizations may not recognize is that the web can be utilized as an 
effective and efficient media for improving organizational performance instead. The current web 
authoring/development software helps them develop fairly sophisticated and highly effective web-based 
performance improvement tools such as web-based database (DB) systems in a reasonable timeframe without 
having to acquire professional programming skills. 

Web-Based DB Systems Can Improve Human Performance 

Some organizations may design web-based DB systems by programming from scratch, which gives 
flexibility in the design. Developing sophisticated web-based DB systems in a short period of time can be a 
challenge to many organizations and they may outsource the development and maintenance, which can be costly. 
Some organizations may use a WYSIWYG (what you see is what you get) web authoring software, which usually 
provides a broad range of built-in, ease-of-use functions. The ‘power’ and 'ease of use' are important factors, but the 
degree of ‘productivity’ is also an important factor for selecting an authoring tool (Fairweather & O’Neal, 1984). It 
is about how much or how sophisticated products you can produce within a certain period of time. 

Frontpage, a product of Microsoft, is one of the web authoring software available for web designers, even 
non-programmers, to use. It is fairly easy to use and moderately powerful, and its productivity is considerably high. 
Frontpage provides a Database Results Wizard, which is a menu-driven, user-friendly environment. It takes users 
through the process of creating or selecting a database record source, choosing specific record fields, specifying 
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criteria for filtering fields, and selecting sort order of selected fields. Within several clicks, by following the steps in 
the Database Results Wizard, users can create a database, generate necessary webpage interface, make a database 
connection between the database and the web interface, and set up webpages to populate data from the database. 
Users are not required to know a programming language to accomplish this task. FrontPage saves the webpages as 
Active Server Pages (ASP), in which necessary Structured Query Language (SQL) statements and VBscript code are 
automatically generated. More advanced web designers who are familiar with using SQL and VBscipt in ASP can 
customize the settings in the Database Results Wizard. They may also design customized ASP pages to develop 
various types of report systems. 

A benefit of using a web-based DB system is that it functions as a centralized depositor of various types of 
information for multiple users at different locations. Users can disseminate new information by entering it into a 
web-based DB system, and update existing information instantly from anywhere at anytime. Upon request and 
within a couple of keystrokes, various types of reports can be generated and displayed to users. Such web-based DB 
systems are essential performance supporting tools for knowledge workers in the 2U' century, who reply heavily on 
using dynamically and fast changing information. 

In the following section, I will describe a case where an in-house development of a web-based student 
enrollment tracking DB system was utilized as a performance improvement tool. I will explain major steps of 
developing such a system. 

A Case Where Web-Based DB Systems Improved Human Performance 

The Subsurface Science Graduate Program (SSGP) is a federal grant based, “collaborative effort by the 
universities of the Inland Northwest Research Alliance (INRA) to enhance research and education in the subsurface 
sciences” (http://ssgp.boisestate.edu/). Eight universities in the northwestern region of the U.S. are participating in 
the SSGP graduate program. Students from the eight participating universities take SSGP distance courses via a 
synchronous videoconferencing system and a course management system. 

Students in each university are registered in their home university’s registration system, and it is difficult 
and time-consuming for the eight universities’ administrators, instructors, and technology- support personnel to 
collect frequently updated student enrollment information from each participating university. Since the program is 
new and fairly small, it is not an option to purchase a learning management system to manage the SSGP graduate 
program’s student enrollment information. If the participating universities do not share their student enrollment 
information with each other in a timely manner, it can create serious problems. For example, 1) some universities 
may fail to schedule the video-conferencing classrooms for their students, 2) students may not be able to receive 
course information from the teaching university on time, 3) instructors may not have information about their 
students, and 4) students may not have access to the course management system. 

The participating universities recognized the performance problem as a communication problem, and 
decided to use a web-based student enrollment tracking DB system to share student enrollment information with 
each other more effectively and efficiently. By using the centralized student enrollment tracking DB system, they 
could enter new student enrollment records, edit student information in existing records, delete obsolete records, or 
search for student information by their name, course title, and home university, from their location, at anytime. 

How the Web-based Student Enrollment Tracking DB System Was Developed 

Frontpage 2002 was used to develop the SSGP’s web-based student enrollment tracking DB system. 
FrontPage’ s Database Results Wizard generated a database editor system, which allowed users to create new records 
and edit or delete existing records (see Figure 1). The data were stored in an Access database in the server. 
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Figure 1: The Database Editor of the Web-Based Student Enrollment Tracking System 

Another important feature of this student enrollment tracking DB system was the course roster report 
system. Since students from eight different universities were participating in the SSGP program, it was critical that 
each university’s administrators be able to obtain course rosters that provided necessary information for supporting 
students as well as instructors. 

A course enrollment status report page was manually designed by using ErontPage’s database results 
feature with specific SQL statements. The report page was designed to allow users to select a course and populate 
the selected course information as well as the course roster (see Eigure 2). In order to calculate the number of 
students currently enrolled in each course, a customized SQL statement was entered during the Database Results 
Wizard steps. The customized SQL statement used to count the number of the records that contained the selected 
course ID number was; 

SELECT Count (courselD) 

EROM Results 

WHERE (CourselD = TCourselD:;'); 
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the number of students 
enrolled in the course 



Figure 2; The Design of the Course Enrollment Status Report Page 

More specifically, when a user selects the course title, SP03-001 Terrestrial Subsurface Processes II from 
the drop-down menu and presses the Submit Query button in this course enrollment status report page, the value 
‘SP03-001’ is being processed by the server-side scripting engine. Then, the system screens through the records of 
the ‘Results’ table in the Access DB and locates records, the CourselD field of which matches the selected value 
‘SP03-001’ (see Figure 3). The system counts the number of records that have matched. Then, the processed result is 
presented in the report ASP page (see Figure 4). 
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Figure 3; The Selected Search Item Is Being Used to Search Records in the Access Database 




Figure 4; The Course Enrollment Status Report Page 

Another feature added in the course enrollment status report page was a hyperlink that allowed users to 
download the report page as an Excel file (see Eigure 5). The FITML code necessary to make a simple hyperlink to 
another ASP file called, CourseStatus_report_excel.asp, would be: <a href="CourseStatus_report_excel.asp">. In 
this case, the hyperlink needed to be set to send the value of the selected course ID (e.g., SP03-001) to the ASP 
page, CourseStatus_report_excel.asp. To do so, VBScript code <%=Request(“courseID”)%> was inserted after the 
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href tag followed by a question mark in the HTML: i.e., 

<a href="CourseStatus_report_excel.asp?courseID=<%=Request("courseID")%>"> (see Figure 6). 




Figure 5: A hyperlink to Another ASP Page 
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Figure 6: VBScript that Sends the Selected Course ID to Another ASP Page 

Then the page, CourseStatus_report_excel.asp, was designed to populate the course roster using the value 
of the selected course ID (e.g., SP03-00I). In this page, the following VBScript was inserted under the <body> tag: 
<% Response. ContentType = "application/vnd.ms-excel" %>. This VBScript was used to present or save the content 
of the ASP page using Excel application. In other words, when users were forwarded to this page, the system would 
activate to either save the page in an Excel document or open the Excel document in their browser depending on the 
users’ choice of action and browser type. 



Summary 

Using recent web authoring software such as ErontPage, non-programmers can develop fairly sophisticated 
web-based database systems in a considerably short timeframe. In developing a web-based DB system, the use of 
server-side scripting is necessary. Server-side scripting such as ASP written in VBScript allows webpages to 
communicate with a database program such as Access stored in the server. A web-based DB system can be designed 
with various SQL statements to manipulate data stored in a database file - i.e., entering new records, editing existing 
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records, and searching for specific records. Such web-based DB systems are excellent performance improvement 
tools for the knowledge workers in the 2T‘ century. They provide dynamically changing information to multiple 
users at distance and help them improve their performance levels. 
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The evaluation of the Garbage bunny: A theoretical evaluation 
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Abstract 

In this contribution a theoretical evaluation study is reported on the use of a pedagogical agent, the 
garbage bunny. This agent operates in an electronic learning environment where pupils learn about sorting garbage 
and recycling. Based on all sorts of information, pupils have to choose the most ecological alternative out of a range 
of products. The garbage bunny is developed to coach students in their problem solving process. 

The evaluation consisted of the design of an optimal agent, based on a task analysis and research findings with 
respect to pedagogical agents. Next, the actual agent was compared to the theoretically ideal agent. Finally a 
comparison was made with three existing agents that also operate in a problem solving environment. The evaluation 
suggests that especially the object, adaptivity and delivery modalities need some optimization. 

Introduction 

Different authors have suggested animated pedagogical agents to be promising with respect to delivering 
adapted support to learners (e.g. Johnson, Rickel & Lester, 2000; Gregoire, Zettlemoyer & Lester, 1999; Baylor, in 
press). Such agents may adapt their support to learning paths of individual students and provide them with non- 
verbal feedback through means of facial expressions and gestures. They are designed to operate in an educational 
setting (Shaw, Johnson, & Ganeshan, 1999). 

The use of open-ended learning environments with ample learner control strengthens the need of 
supporting learners in their task execution. Students are confronted with a complex or ill-structured problem that has 
to be looked at from different perspectives to generate a suitable solution (Spiro, Feltovich, Jacobson & Coulson, 
1991). Given the complexity of the problem, students will need support on different levels, not only on the level of 
the content, but also on a problem solving and metacognitive level. Moreover, research in the aptitude treatment 
interaction tradition reveals a strong interaction between learner characteristics and instructional interventions 
(Snow, 1986; Snow & Swanson, 1992). For instance, the required amount of support depends on specific student 
characteristics such as prior knowledge (Ross, Rakow & Bush, 1980). Additionally, Clark (1991) demonstrated that 
both too much and insufficient support might be detrimental. Thanks to their potential to adapt support to learners, 
pedagogical agents seem to offer an opportunity for providing support to students when solving problems. 

Sorting garbage and recycling can be defined as an ill-structured problem. Problems in this domain have to 
be looked at from different perspectives; not only the garbage itself, but also the production process, the basic 
products, the recycling process, and issues such as comfort for the users have to be considered. In this contribution 
an evaluation is reported with an electronic learning environment to help students learn about recycling and waste 
prevention. Since this environment, was developed as an individual learning environment, not necessarily used 
under supervision of a teacher, a coach was integrated to assist learners in their problem solving process. This 
pedagogical agent will be referred to as the ‘Garbage bunny’ . 

This contribution is a theoretical review (Karat, 1997) of the garbage bunny. 

Method 

The evaluation focuses on the pedagogical agent in the software program. More specifically, support 
provided by the agent will be looked at. To do so, the categorization scheme developed by Clarebout, Elen, Johnson 
and Shaw (2002) is used. The different aspects of this categorization scheme are: 

Object of support: support can be directed towards different aspects of the learning environment, i.e. content, 
problem solving, metacognition and handling technology. 

Learner control with respect to support: Pedagogical agents can control the support and take the initiative to 
provide support or students themselves may take this decision and request support. An open learning environment 
by definition implies a great deal of learner control (Hannafin, 1995). However, since students seem not always 
capable of adequately choosing for themselves (Large, 1996; Williams 1996), and given that reviews on learner 
control do not reveal a consistent positive effect (e.g. Goforth, 1995), this aspect requires further consideration. 
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Adaptability of support: The required amount of support depends on students’ prior knowledge (Ross, Rakow & 
Bush, 1980) and other learner characteristics such as students ‘aptitude and metacognitive skills (Clark, 1990). Clark 
distinguishes between three levels of support: a) no support (only goal statement and information are provided), b) 
activation (goal statements, information and indications on how to process the information are provided) and c) 
compensation (instruction takes the cognitive burden, cognitive processing over from the learner. A gradual 
withdrawal of support is generally advocated (Collins, Brown & Newman, 1989; Jonassen, 1996; Vermunt, 1992). 
However, not only the amount or quantity of support can be adapted to the needs of students, but also the object of 
support, they may need support on a domain-specific level or on a metacognitive level for example. Either the 
program or the learner can determine the change in quantity and object of support. 

Delivery modalities. The pedagogical agent can entail different delivery modalities, which might affect the effect of 
pedagogical agents on learning (Moreno, Mayer, & Lester, 2000). Pedagogical agents communicate verbally or non 
verbally. Reeves and Nass (1996) indicated that students view the interaction with a computer (pedagogical agent) 
as a social interaction, in which three features are essential namely, a) image, b) voice and c) personalized language. 
A pedagogical agent has by definition, an image. Hence this distinction is not included in the typology. However, 
the other two aspects are. With respect to verbal language pedagogical agents, can use either text or speech to 
communicate with the learner. Examples of nonverbal language are head nods and gestures to provide feedback. The 
pedagogical agent can communicate in a personal way and engage in dialogue or in a less personal way, namely 
through a monologue. 

Timing of support: A pedagogical agent can deliver support either up front, by presenting for instance at the start 
of the interaction information the student will need to solve a task, or, just-in-time, during the execution of the task. 

It might also be that the agent provides delayed support by giving for example information about the process after 
task completion. 

Support style: This dimension refers to the different roles and modalities of an instructional agent. Based on a 
literature, six typical roles for pedagogical agents are identified: 

Supplanting: the instructional agent assumes responsibility for the tasks and performs them 
for the learner. The learner observes the instructional agents while he performs the task (e.g. 
Salomon, 1994). This can be compared to what Clark (1990) called a compensation where 
(meta)cognitive activities are taken over for the learner. 

Scaffolding: the instructional agent performs those parts of the task that learners are not yet 
able to perform themselves (Collins et ak, 1989, Jonassen, 1996). This might allow learners to 
perform on a level just above their current level, in their “zone of proximal development” 
(Vygotsky, in De Corte, Verschaffel & Lowyck, 1994); 

Demonstrating: the instructional agent shows how a task is performed after which they 
observe how the learner performs the task (Merrill, 1994); 

Modeling: the instructional agent shows how a task is performed while revealing and 
explaining his reasoning process. The instructional agent solves a task while he articulates 
how problems are solved, what strategies are used, and what mental model are necessary to 
understand the task (Jonassen, 1996) 

Coaching: the instructional agent provides hints and feedback, and activates the learner when 
the learner is performing the task. The instructional agent observes the learners when they are 
solving a task and provides guidance when students experience difficulties (Barab & Duffy, 
2000) and 

Testing: the instructional agent tests the learners’ knowledge about certain aspects of the task 
to guide the learning process (Martens & Dochy, 1997). 

These different roles reveal four different modalities of support: a) executing, b) showing, c) explaining and 
d) questioning. Each of the above-described roles can be characterized by (a combination of) analytical modalities. 
Eor instance, if the role of a pedagogical agent is coaching, support can consist of a combination of explaining and 
questioning. 
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Figure 1 provides an overview of the categorization scheme. 




A theoretical evaluation of the pedagogical agent allows for detecting some weaknesses that still can be 
adopted before finalizing the product. Based on this categorization scheme an ‘optimal’ agent will be constructed. 
To construct this ‘optimal’ agent, use will be made of research findings relating to the effect of certain agent 
characteristics on learning. 

Additionally the Garbage bunny will be compared to existing pedagogical agents that are also used in a 
learning environment with ill-structured problems (e.g., Adele, Steve, Herman the Bug). 

Program description 

The garbage program offers an environment in which students can develop their ecological awareness and 
skills by solving specific problems such as selecting the most ecological school materials. 

Students enter a virtual school in which they are asked to select the most ecological product from different 
alternatives. They can click on different rooms: the lunchroom, classroom, and playground. In each of these rooms 
they are confronted with specific products that are often used, for instance pencils, glue, lunch packages. When 
clicking on a certain product, they receive information on that product, going from its raw materials, package, cost 
to comfort. In order to make their choice for the most ecological alternative, they have to consider not only the 
product itself, but the whole production process, comfort and prices. When students think they have collected 
enough information to select the most ecological product, they can do a test. In this test, different questions are 
asked on the different categories (comfort, production process, etc.) and the final question is “Which product is the 
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most ecological?”. When they give a “wrong” answer, students receive feedback on why their answer was wrong 
and are encouraged to continue their investigations. This also means that they can go back to the information 
section. 



Evaluation 

Optimal agent 

Given the ill-structuredness of the task, students will not only need content information to solve it, but they 
will also need some problem solving and metacognitive skills to reach an adequate solution. As such, it can be 
expected that students will need support on these different levels. Support on a content level is provided by the 
program itself, students can access additional information by clicking on the products. The program does not 
provide support on a problem solving and metacognitive level. Consequently, it is expected that the agent will direct 
its support on those two levels. It is expected that the agent will coach the learners by explaining and asking 
questions. Additionally the agent might also model the problem solving process. Research findings show that 
students do not always make adequate choices with respect to their learning process (e.g., Clark, 1991; Large, 1996; 
Hill & Hannafin, 2001). Shaw et al. (1999) revealed that students prefer to request support of the agent, rather than 
automatically receiving it, but that they do not use it. Consequently, it seems advocated that the agent controls the 
delivery of adaptive support. 

Most research with pedagogical agent has studied modality effects of agents. Different studies by Moreno 
and colleagues (Moreno, Mayer & Lester, 2000, Moreno & Mayer, 2000; Moreno, Mayer, Spires & Lester, 2001) 
showed that an agent using spoken personalized language leads to the best results. Atkinson and Mayer’s 
experiments (submitted) revealed that a human voice resulted in better performance than a machine synthesized 
voice. In her research, Baylor (in press) focused on the comparison between a fully animated agent, a static agent 
and no agent. Apparently the fully animated agent was found to be more engaging, more person-like and to have a 
higher degree of credibility. However, the effect on learning was not studied. Concerning the timing of support 
delivered it is important that the agent delivers the support just in time, van Merrienboer distinguishes between 
information relevant to recurrent aspects (rules, procedures, prerequisites) and to non-recurrent skills (system, 
approaches and heuristics). Support for recurrent aspects should consist of just in time information and immediate 
feedback on the quality of performance of certain skills, while support for non-recurrent aspects should consist of 
both information presented up front and delayed. 

Based on these findings the following theoretical optimal scheme of the Garbage bunny can be drawn: 
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Figure 2: An optimal agent 



Figure 3 presents the analysis of the Garbage bunny. 
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Figure 3: The Garbage bunny 

Figure 3 reveals the following observations with respect to the different aspects of support: 



Object of support: the pedagogical agent provides indeed support on problem solving level. When students enter a 
room or a new part in the program, the agent explains how they have to start with their problem. No support is 
provided on content level. This corresponds to the expectations, but in contrast to what was expected no 
metacognitive support is provided. 

Learner control with respect to support: the agent determines itself when and what support is provided. The student 
cannot interact with the agent, cannot request for additional support or refuse additional support. 

Adaptability of support: the Garbage bunny fixes the amount and object of support, no differentiation is made 
between students with different needs; the learning paths of the students are not taken into account. Moreover, if 
students return to a previous section or room, the agent will simply repeat what he said the first time. 

Delivery modalities: The agent communicates verbally and non-verbally. Every time the agent says something 
through speech, the text of what he says is displayed as well. This written text remains on the screen until the learner 
starts a new action. This is in contrast to the expectations. 

Timing of support: The Garbage bunny provides just-in-time support. It was expected that based on the kind of skills 
needed to solve the problem some information would have been provided in advance, that for the non-recurrent 
skills. 



Ill 





Support style: The Garbage bunny introduces the problem when students enter the virtual school He acts as a coach 
by providing guidelines to solve the problem and to work in the environment. He explains how to work and what to 
do. While in the ‘optimal’ agent the possibility was uttered that this agent would also model the problem solving 
process, this is not the case for the actual Garbage bunny. 

Table 1 compares the Garbage bunny with three other previously analyzed (see Clarebout et ah, 2002) 
agents that also serve as coaches in problem solving processes. The first agent is Adele (Shaw et ah, 1999; Johnson 
et al. 2000, Ganeshan, Johnson, Shaw & Wood, 2000). Adele supports distance education in the domain of medicine 
and dentistry. She exists in three interaction modes. In the advisor mode she observes the student when performing 
an action. If this action is consistent with standard practice, Adele will interrupt the students and suggest another 
action. In the practice mode, Adele provides advice upon request of the student. In this mode, Adele does not 
interrupt the learner. A final mode is the examination mode. In this mode, Adele does not give advice during the 
task, but provides feedback after task completion. 

Steve (Johnson et al., 2000) is an acronym for Soar Training Expert for Virtual Environments. He 
demonstrates “skills” involved in a specific task in a virtual environment. The student can walk around in the 
environment and look at Steve’s demonstration from different angles. The student can interrupt the agent and ask 
him to complete the task. When the students are performing the task themselves, they can ask Steve to show them 
what to do next. While Steve demonstrates part of the task he explains what he is actually doing. 

Herman the Bug (Lester, Stone & Stelling, 1999). Herman is an alien with human-like movements and 
facial expressions. Herman inhabits a learning environment called “design-a-plant” for the domain of botanical 
anatomy of physiology. He offers advice about the relation between plant features and environmental features, 
encouragement when students encounter difficulties and feedback on their choices while designing a plant. 



Table 1: The Garbage bunny compared to other pedagogical agents 

Pedagogical agents 





Garbage bunny 


Adele 


Steve 


Herman the Bug 


Modality 


Explaining 


Explaining/questioning 


Showing/ Explaining 


Explaining 


Role 


Coaching 


Coaching 


Demonstrating/Modeling/coa 

ching 


Coaching 


Object 


Problem solving 


Content/problem 
solving/ technology 


Content/problem solving/ 
technology 


Problem solving/ 
content 


Control 


Agent 


Agent/leamer 


Agent 


Agent 


Adaptability 


None 


Quantity and object 


Quantity and object 


Quantity and object 


Delivery 

modalities 


Spoken / written 
Personalized 


Spoken/written 

personalized 


Spoken/ personalized 


Spoken/personalized 


Timing 


During 


During 


Prior / during 


During 



The table indicates that for most characteristics the Garbage bunny is comparable to the similar agents. The 
different agents take the role of a coach by explaining different issues to the learner. They all provide support on a 
problem solving level, although the other agents provide additional support on a content and technology level. 
Similar to the Garbage bunny none of the others give support on a metacognitive level. The most striking difference 
relates to the adaptability of support. While the other agents adapt their support to what students are actually doing, 
the Garbage bunny does not. 



Conclusion and discussion 

This theoretical evaluation started with the design of an ‘optimal’ agent based on a task analysis and 
research findings with respect to pedagogical agents. The evaluation itself shows that the Garbage bunny deviates on 
three aspects from this optimal agent. A first aspect relates to the object of support. The Garbage bunny does provide 
support on a problem solving level, and not on a content level. However, no support was provided on a 
metacognitive level. While given the ill-structuredness of the task, it was expected that students would need 
guidance with respect to this aspects. Supporting students’ on a metacognitive level would mean helping them in 
planning and regulating their own learning process. Apparently, this seems to be a level of support that is ignored in 
most agent-related applications. The comparison with the three other agents that also operate in a problem solving 
learning environment, reveals similar results. A second point relates to the adaptability of support. Although Clark 
(1991) indicated that adapted support (both too much and insufficient support) might be detrimental, the support of 
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the Garbage bunny is not adaptive. Even worse, the same support is provided every time when students return to the 
same point in the program. Further investigation might be interesting to determine whether this causes additional 
extraneous cognitive load (Sweller, 1988) or whether students just ignore it and continue working. A third difference 
with the optimal agent relates to the delivery modalities. Research results (Moreno & Mayer, 2000; Moreno et al., 
2000; Moreno, Mayer, Spires & Lester, 2001) indicate that the use of spoken text leads to the best learning results in 
comparison to written text alone or a combination of written and spoken text. The Garbage bunny combines written 
and spoken text. Additionally, the written text of the agent remains until the agent starts to talk again. 

Based on this theoretical evaluation suggestions can already be made for optimizing the pedagogical agent, 
such as using only spoken text to provide support, including support on a metacognitive level and adapting the 
support to the learners. However, questions remain with respect to the effectiveness of this agent; 

Does the Garbage bunny help students solving the problem? 

Does the fact that the Garbage bunny is continuously present effects learning? 

Does the remaining of the written text hampers students learning? 

An empirical study might provide insight in these issues. Some of this research issues demand the same 
application without an agent in order to compare the effects of the agent. This relates to a broader problem with the 
development of educational technology. Programs are being developed and released using the latest technology 
before they are actually tested. This is also the case with the Garbage bunny-program. Use is made of the latest 
technology, without any test phase. It reflects the gap between the pace of evolution in technology and research with 
respect to this technology. Often educational software is developed with the latest technology before actually asking 
questions whether this works, whether this has a positive influence on learning and for whom; without waiting for a 
firm research basis to justify the use of these technologies for learning. 
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Abstract 

The complicated issue of student retention in distance education (DE) courses has become a stumbling 
block for the success of the field as a whole. As few as 40% of enrolled students complete DE courses according to 
some studies. Is this the result of inexperienced faculty? Or the different perspective of the adult online student? 
Or, as some claim, the result of the isolated nature of distance education itself? A hierarchy of retention strategies 
is presented based on the retention experience in a nationally acclaimed online education master’s program. Using 
a combination of course design principles and a proactive, mentoring and service minded faculty a completion rate 
of 94% was achieved over a 2-year period. 

Now that distance education is becoming more accepted in mainstream education the retention of students 
is evolving as one of the critical issues in the field. Until the patterns of retention and student withdrawal can be 
identified and addressed distance education will not meet its goals of reaching a greater number of students (Brady, 
2001). At present no national statistics on retention in individual courses or by programs exists. However, the 
majority of available anecdotal evidence points to completion rates in distance education that are consistently lower 
than those of traditional face-to-face programs (Carr, 2000). 

The high withdrawal rate has become an unquestioned norm. A review of ten recent studies of retention 
and withdrawal shows a huge range of 40% to 91% of students completing distance education classes. These same 
studies cited higher completion rates of between 65% and 95% in comparable face-to-face classes. In one study the 
gap between distance education and face-to-face classes was only 4%. At the other extreme, a 26% difference was 
documented between the distance education students and the traditional track (Frew & Weber, 1995). 




□ Low 

□ High 



Meta Analysis of Students Completing Courses Distance Education vs. Face-to-Face 

Professional Teacher Competencies Workshops offered through The University of Northern Iowa (UNI) 
did not experience this kind of normative high attrition. The five courses in the workshop series are based on the 
University of Northern Iowa Professional Teacher Standards and follow the design model employed by UNI in the 
development of over 30 online professional development courses. Key design elements included consistent web 
page design, thorough instructional design, and collaborative learning elements in combination with the active 
mentoring of an experienced faculty. Most importantly the mentoring and monitoring of student performance 
created an expectation of course completion and attrition rates have been held below six percent for over five years 
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Causative Analysis 

Most of the studies in the distance education and adult education literature describe one of two different 
causes of the poor completion rates. One theory proposes that the withdrawals are mostly due to the life 
circumstances. Adult students who are attracted to distance education are older, more likely to be employed and 
married. These students who are juggling full time jobs and family responsibilities withdraw from classes to attend 
to other areas of their lives (Scalese, 2001). Under this approach dropping a course is seen as somewhat outside of 
the control of the university and therefore cannot be addressed through policy. Indeed in adult education students 
opting out and returning to coursework is seen as the more normal pattern. Adult students leave course work when 
their needs are met. Under this scenario dropping of a course should not be seen as a failure (Kerka, ; Tresman, 
2002). The second most common explanation in the literature centers on the nature of distance education and 
blames the high attrition rates on the lack of social contact between the instructor and the students (Faust, 1999; 
Muirhead, 2000). 

The literature supports the idea then that most often the decision to drop is outside of the scope of influence 
of the university. That is to say that most attrition will be impossible to prevent. Distance education can 
accommodate the role conflict that occurs for students by offering flexibility in course work when family and work 
balance issues occur, but under the first scenario this accommodation will not remedy the cause. Even if we accept 
prima facie that we cannot fully resolve the work-family-school balance problem, the difficulties of social isolation 
in distance education is a factor that can and should be mitigated if distance education is to thrive. The separation of 
the instructor and the learner is prima facie part of the definition of distance education. However this separation 
need not give rise to isolation (Keegan, 1996). 

Workshop Design and Implementation to address Isolation 

There are two important arenas in which to address this isolation; first is in the course design and later in 
the implementation of the instruction. Several design elements decrease isolation and thereby promote retention. 
Issues of individual enrollment and asynchronous vs. synchronous work must balance convenience against the 
promotion of retention. The workshops are group-paced rather than self paced. Following a modular approach and 
discouraging students from working ahead helps to cement the learning community. There is social support created 
as the groups progressed through the modules together. So while self-pacing is attractive to students there is a 
trade-off in lower social cohesion. Another issue in the design was the inclusion of synchronous activities. While 
research has shown synchronous communication to increase social cohesion the asynchronous design was a matter 
of practicality since UNI workshops typically enroll 12 to 18 students across four or more time zones. 

Design must carefully support the implementation so one of the first tasks in the design of any of the UNI 
workshops is to fast track the formation of a learning community and promote collaboration among its members. 

The formation of a learning communities are seen as a hedge against withdrawals. The learning community is so key 
to online education that Palloff and Pratt are quoted as saying “The learning community is the vehicle through which 
learning occurs online." (Parloff & Pratt, 1999). In the UNI workshops the development of the learning community 
starts immediately with an icebreaker activity. This email-based activity is designed to draw attention to the 
common human bonds between the students. 

One icebreaker question asks if they could be doing anything at this moment what would they be doing? 
Students are encouraged to respond to their classmates’ postings. Two approaches have been suggested as 
appropriate for the instructor in such a group. Instructor’s can hang back from participating and allow the group to 
form on its own (Jeong, 1996; Jordan, 1999) Alternately the instructor can participate as any other member of the 
group modeling icebreaker postings and commenting on common ground between students for example, if students 
are involved in the same curricular area (Piburn & Middletown, 1997; Seale & Cann, 2000). The UNI experience 
suggests that the involved instructor who models collaborative behavior is a key to retention. The instructor must 
set an empathetic tone and the expectation that this is to be a community. To this end the first few days of the 
workshop are purposefully and exclusively devoted to these activities. 

Research suggests that online groups need to process through similar stages as face-to-face groups. 
Tuckman’s model of forming, storming, norming and performing has been postulated as applicable in the online 
world (Glover, 2003). Each stage demands an outlet within the learning community. Just as the icebreaker sets the 
expectation for forming inclusion of collaborative activities permit the group(s) to establish norms. One to two 
collaborative activities are included in each module of the workshops. For each of these assignments different 
partners are assigned so that by the end of the workshop the students have had a chance to work with all of their 
colleagues. The collaborative assignments are included as a significant portion of the student’s grade in order to 
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builds accountability to the learning community. Once again the instructor sets the expectations for collaboration by 
explicitly directing the pairings, ensuring easy access to each other through a class roster and by listing alternate 
email addresses for students and setting interim deadlines. A module in the workshop may last 10 days but the 
instructor will ask for a progress report on the group project after four or five. Instructors may assign group leaders 
during the first module and thereafter leave it to the group to rotate the leadership. This would facilitate norming 
and performing. The expectation of checking in with your group members even if it is just to say “I haven’t had 
time to read the assignment yet” is also explicit. 

In the UNI experience groups do not seem to storm as such. Rather assignments are left to the last minute 
and approximately 20% of the time handed in as individual assignments due to a group member’ s non- 
responsiveness. When this occurs the instructor investigates in a curious, not furious way. The assumption is that 
some unavoidable life circumstance has interfered with the student’s good intentions to complete the assignment. 
Students must be coached to also give the “benefit of the doubt” in this way. When a student has defaulted on their 
responsibility to the group the instructor gives quick feedback and monitors more closely to ensure this does not 
occur again. It is common to place the defaulting student in a leadership role for the next collaborative assignment. 
Often the burden of leadership creates the social pressure not to default in the future. The instructor also gives quick 
responses to the student who brought the defaulting to the instructor’s attention. These responses always start with 
acknowledgement of the frustration and the difficulty and thank the student for having the courage to speak up. The 
offending student’s confidentiality is maintained but the reporting student is informed that action has been taken. 
This is again respectful of the norms of the learning community that are based on individual responsibility. 

Constructivist activities 

The collaborative assignments are all based on applying theoretical principles to real world 
teaching scenarios. Students are all active practitioners involved in the classroom who can draw upon the wealth of 
their teaching experiences to illustrate various theoretical points. Each workshop includes the development of a unit 
of instruction. The assignments take a production-style approach to developing this unit so that a piece of it is 
assigned in each unit until the student has a finished lesson plan ready to be implemented and infused with 
technology. Student evaluations of the workshops highlight this design approach as a key factor in maintaining their 
interest in the course and their full participation. As with face-to-face adult students, online adult students want 
tangible problem solving skills that they can apply to their professional situations immediately. 

The Relationship Between Instructor and Student 

Assuming that the quality of social interaction is greater with experienced instructors and in keeping with 
the recommendations in the literature the quality and commitment of the instructors was recognized as a 
contributing factor in controlling withdrawals. Instructors with hundreds of hours of online experiencewere 
recruited and the timeliness of email responses was a discriminating factor in the selection process. More than one 
qualified instructor was not considered if emails sent on an “acquaintance basis” went unanswered. 

A quick response via email was an indicator of a service mentality and an eagerness to engage in online 
interpersonal connections. Two components of service to students evolved during the instructional phase of these 
classes. First is the ability to do what Dirksen and Leggett (2002) refer to as cybemag and cyberschmooze (Dirksen 
& Leggett, 2002). A delicate balance between the two approaches must be achieved. Too much nagging and the 
student is alienated, too much social banter and the instructor’s credibility can be called into question. Yet the 
social support from an instructor can be a key factor in a student completing the workshop. Instructors function in 
an advisory capacity on occasions when a student is considering dropping the workshop. A high level of service 
mentality is employed to accommodate the individual needs of the student while preserving the rigor of the 
coursework (Ainslie & Eves, 1998; Willis, 1992). 

A Mentor and a Monitor 

In addition to cultivating a socially supportive and instructional relationship with the students instructors 
take an active role in monitoring student progress. A few days after the conclusion of each module the instructor 
reviews an online grade book and monitors students who were missing assignments. Proactive emails and telephone 
calls are made to help students keep on track with their workload and to investigate any sources of confusion with 
respect to the material (Bennett, 1997). Other research has found this active phone approach effective. Towles 
found that 55% of the DE students in his study who receive a mentoring phone call successfully completed their 
course while 64% of those not called withdrew (Towles, 1993). Most often what these contacts revealed was some 
outside responsibility that was draining time away from course work. This anecdotal finding concurs with the 
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literature that states that family/work responsibilities were most often the cause of dropping a course. Instructors in 
the UNI courses were given the authority to make individual accommodations. For example late submissions were 
accepted. In some cases minor assignments were made optional as a retention method. 

Ultimately an instructional design was in place that supported the formation of a learning community and 
through its success, would promote retention. Implementing this relied on two principles first, that quick responses 
from the instructors would be essential. A 24-48 hour turn-around on student work was set as the standard. This 
was done to provide the students with timely feedback but also to combat the sense of isolation distance learners are 
documented as feeling. 

Each instructor response would follow a praise/question/polish approach. First some specific praise for the 
submitted assignment would be detailed. Any areas in need of clarification or rewriting would be addressed in the 
form of a question rather than a statement of deficiency. Additions or minor improvements would then be 
suggested. This format was followed in order to ensure that positive feedback was the main emphasis of 
instructor/student contact. This kind of positive tenor would serve to motivate students. It was especially important 
to respond to low quality work in this framework. All submissions that fell below the standards of excellent as 
described in the grading rubrics were responded to following this coaching model. The students were then offered 
the chance to improve their grade by polishing the submission and resubmitting it. A kind of continuous quality 
improvement approach evolved and all work was considered a work in progress. Occasionally, students would even 
go back several modules and rework graded submissions based on new knowledge they had gained (Schoenfeld- 
Tacher & Persichitte, 2000). 

One unique aspect of the instructor/student relationship in the UNI courses was seen in that the instructor 
focused on more than just grading the assignments. A secondary goal of the email exchanges was to start a dialogue 
with the student. This dialogue which was similar to the didactic conversation described by several researchers. 
(Dillon & Walsh, 1992; Holmberg, 1995). Dialogue was often based upon ideas that the student’s submission had 
sparked in the instructor’s thinking. These comments took the form of ‘By the way’ (BTW) additions to the main 
task at hand. Side conversations were generated and off-topic resources were shared frequently. One student for 
example, provided an in depth look at layered curriculum in response to a BTW comment from the instructor. On 
another occasion several students advised the instructor on a classroom management problem she was experiencing 
and in almost every class students debated the issue of class size as a barrier to active learning activities. 

Student Feedback 

Mid course and end of course comments consistently documents that students found the flexibility of online 
learning a key factor in their decision to enroll in a workshop and to complete it. When students made additional 
comments in an extensive summative evaluation of the workshop often they praised the timely response of the 
instructors. The Online programs at UNI involve web based readings, journaling, video reviews and collaborative 
projects. The success of the programs has been marked not only by the high completion rate but also be student and 
institutional reviews. 



Conclusions 

UNI Professional Teacher Competencies Workshops have maintained a stellar retention rate of less than 
6% for more than five years. This is due to design and implementation elements in the courses. Instructors act as 
proactive, socially supportive mentors in that they model good online behavior, consistently offer 24 hour turn 
around on assignments, coach individual students in completing constructivist based group projects, assist groups in 
forming, norming, storming and set high expectations for group performance. Instructors actively monitor student 
progress and inquire immediately if deadlines are not met. These inquiries are couched in a professional, respectful 
tone that gives the student the “benefit of the doubt” in that the instructor assumes that students are committed to 
completing the workshop. The instructor’s tone is collegial ensuring that students receive positively voiced feedback 
and peer support. “By the Way” commentary enhances the workshop and helps students to solve instructional 
problems in the real world. 
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Abstract 

Business needs in multinational corporations call for learning outcomes that involve problem solutions and 
creating and sharing new knowledge within workplace situations that may involve participants in varying national 
settings. Computer-supported collaborative learning (CSCL) applied in courses that emphasize work-based activities 
is an appropriate tool for such learning situations. A theoretical model for WBA-CSCL (work-based activities and 
CSCL) is derived and from it elements of a pedagogical model are discussed. Finally, issues and challenges arising 
during implementation of the framework and the model are identified. 

Introduction and Problem Statement 

In the workplaces of professionals in multinational corporations problems and challenges continually arise 
that involve creating new solutions and constructing new knowledge, and indirectly involve improving 
communication and understanding among colleagues in different parts of the world, working virtually with one 
another. Key strategies involve capturing and sharing the tacit knowledge in the corporation, a particularly 
challenging task in that such knowledge is often difficult to see and express, is personal, and involves subjective 
perception, intuition, and foresight (Nonaka & Konno, 1999). Learning takes place in the context of these challenges 
and strategies. Such learning can be either formal such as via participation in structured courses, or informal, such 
as via mentoring and coaching in the workplace related to cognitive apprenticeship (Collins, Brown, & Newman, 
1989); self-directed processes and experiential learning (Brookfield, 1995); and participation in "learning 
communities" (Wenger, 1998). 

Both formal and informal approaches have their limitations as well as strengths. Among the strengths of 
structured learning are guided opportunities to learn while interacting with a new set of peers, broadening the 
learner's contacts beyond his or her workplace colleagues. Among the strengths of informal learning in the 
workplace is the anchoring of learning in problems and situations that are real and relevant to the learner (Billettt, 
2001). Collaborative learning within courses that emphasize authentic work-based activities integrates both these 
sets of strengths and thus has strong applications in both formal and informal learning for professionals in the 
multinational corporate setting (Lim, Tan, & Klimas, 2001). In this paper we propose a form of formal learning 
emphasizing computer- supported collaborative learning applied to work-based activities and integrating the 
strengths of informal learning as a way to construct the new knowledge and new solutions necessary in the corporate 
setting. 

The questions addressed in this paper are: 

• Why is computer-supported collaborative learning (CSCL) involving work-based activities 
(WBA) an appropriate response to key needs in a multinational organization? 

• What is a theoretical framework for WBA-CSCL in multinational corporate settings? 

• What is a pedagogical model for concrete realizations of WBA-CSCL? 

• What issues and challenges confront the realization in practice? 

First, an analysis relating to the appropriateness of CSCL involving work-based activities for constructing 
new knowledge and creating solutions to business challenges is given. Then a theoretical framework for WBA- 
CSCL is proposed, derived from Gifford and Enyedy's "Activity Centered Design" (1999). Following this, the 
framework is used in defining a pedagogical model for WBA-CSCL. A discussion of implementation challenges 
and issues for further research based on the application of WBA-CSCL in courses at the Shell Open University 
concludes the paper. 



Learning in a Multinational Corporation 

This section addresses the question of why computer-supported collaborative learning involving work- 
based activities is an appropriate response to key needs in a multinational organization. The discussion focuses on 
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the importance for corporate learning to be directly relevant to business and workplace needs and elaborates on the 
rationale for merging formal and informal forms of learning through courses where work-based activities involving 
collaborative learning are the focus. The need for a theoretical framework and a pedagogical model for this kind of 
learning will be highlighted. 



Learning related to business needs 

Organizational expectations related to corporate learning are focused on changes in performance that affect 
the business results. However, organizations often do not see a direct relationship between business results and their 
investments in formal corporate learning because much traditional formal learning is limited in its transfer to the 
daily workplace (Smith, 2002). Partly this is because formal courses too often focus on prepositional or conceptual 
knowledge (‘knowledge that’) and too little focus on procedural knowledge (‘knowledge how’) (Billettt, 2001). In 
contrast to knowledge transfer, corporate learning should be characterized by knowledge sharing, capturing 
experiences, reusing them, creating new knowledge, and recognizing and solving workplace problems, in a process- 
oriented, collaborative manner (for a literature review, see Collis & Margaryan, 2003). As an example, at a recent 
EAGE (European Association of Geoscientists and Engineers) conference, the petroleum industry’s current needs 
were noted as including (a) the necessity for professionals to understand different disciplines rather than only their 
own discipline in isolation, (b) the development of teamwork, and (c) the necessity to understand evolving 
technology with a focus on the general principles of how a range of different technologies function rather than 
specifics of only a few types (Vedrenne, 2002). All of these call for meta-cognitive capabilities that are best 
fostered in collaborative learning situations focusing on work-based situations rather than courses characterized by 
knowledge transfer (Seufert & Seufert, 1999). 

When supported by network technology, courses based on collaborative learning centered around real 
workplace problems and opportunities can create an environment where individuals and groups can jointly generate 
new ideas and get feedback on their own ideas from peers while working in complex learning situations in their own 
workplaces. The interactions can provide possibilities for individuals to articulate their own tacit knowledge and 
share it with others. Collaboration occurs in a variety of ways: among the course participants, within a team sharing 
the same workplace, or between teams with genetically similar problems but different workplaces. Within the 
workplace, collaboration also involves one's peers and also one's supervisor and relevant others in the corporation. 
All these need to be integrated by the course instructor so that sharing occurs. This approach to CSCE involves the 
use of networked technology that includes groupware tools such as shared archives, and tools for structuring, 
monitoring, and motivating overall course processes. CSCE is facilitated by a “computer-based network system that 
supports group work in a common task and provides a shared interface for groups to work with” (Hsiao, 2003). 
CSCE can build upon the affordances of computer-supported systems that can support and facilitate group learning 
in ways that are not achievable in traditional classroom-learning environments (though CSCE should not necessarily 
replace face-to-face contacts). 

This paper argues that an integration of formal and informal workplace learning via courses with work- 
based activities involving computer-supported collaboration as the integrating element is an appropriate way to 
address the business needs of a multinational corporation. Such an integration can be called WBA-CSCL (work- 
based activities and computer-support collaborative learning). Eirst a theoretical framework is needed as a 
conceptual basis for such integration. In addition, a pedagogic model is important in order to operationalise the 
conceptual framework in practice. In the following section, a theoretical framework for WBA-CSCL will be 
discussed, and following that a pedagogical model based on that framework will be proposed. 

Towards a Theoretical Framework for WBA-CSCL 

In the first section, tthe social-cultural Activity Theory will be discussed as the basis for a theoretical 
framework for learning based on work-based activities and collaboration carried out in computer-supported 
collaborative learning (CSCE) environments. The key components of Activity Theory in relation to corporate 
learning will be elaborated in the following section and the WBA-CSCL Eramework will be introduced. 

Activity Theory and CSCL 

Activity Theory (Leont'ev, 1978; Vygotsky, 1978, originally published in Russian in 1930) is being 
increasingly used to understand social aspects of technology-supported learning (see for example Jonassen, 2002). 

In the design of technology-supported learning in the corporate setting, social aspects of learning have often been 
neglected, with the design efforts primarily focused on either the content of the learning domain or profiling of the 
individual learner (Gifford & Enyedy, 1999; Russell, 2002). However, the social aspects of learning are particularly 
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important in work-based learning focused on workplace practices and problems, the needs of the business, working 
with others, complex interdependencies, participation, and engagement (Billett, 2002). 

Activity Theory can contribute to the understanding of computer-supported collaborative learning (CSCL) 
in the corporate setting by “[understanding] learning not as the internalization of discrete information or skills by 
individuals, but rather as expanding involvement over time - social as well as intellectual - with other people and 
the tools available in their culture” (Russell, 2002, p. 65). The key propositions of Activity Theory relevant for 
WBA-CSCL are that human activities are the unit of analysis, and they are always situated in a context or "an 
activity system" with seven main elements (“contextualized activity”; Engestroem, 1987, Kuutti, 1996, and Gifford 
and Enyedy, 1999) as shown in Eigure 1: 



Instruments 




Eigure 1. An activity system (Engestroem, 1987) 

In Eigure 1, the seven main concepts relate to: 

1. Subject: The actors involved in the activity 

2. Instruments: Tools, theories, and procedures that mediate the activity 

3. Object: Products acted upon by the subjects during the activity 

4. Community: Socio-cultural context in which the activity takes place 

5. Rules: Implicit and explicit rules and norms of the community that constrain the activity 

6. Division of Labor: Horizontal and vertical roles and relationships within the community that affect 
task division 

7. Outcome: Transformation of the objects; the overall intention of the activity system (Jonassen, 

2002). 

The following section will discuss in more detail how Activity Theory can be adapted to to the particular 
type of corporate learning represented in WBA-CSCL. 

The Work-Based Activities-CSCL Framework (WBA-CSCL) 

An activity system for CSCL settings in courses involving work-based activities in multinational corporate 
settings as shown in Eigure 2. 








Figure 2. An activity system representation of WB A-CSCL in a multinational corporate setting 



The elements can be described as follows: 

Outcome: Business impact through (a) increased knowledge sharing particularly among colleagues in 
different locations and from different national cultures, (b) solutions to immediate problems in the workplace, and 
(c) competence development of individuals based on performance. 

Objects: Workplace problems as learning tasks. Oliver and Herrington (2001) identify a variety of activities 
appropriate for task-based learning, including inquiry tasks, projects, and investigations (p. 31). What is critical in 
their view of tasks is that all involve authentic problems. Tasks are open-ended, not simple but reflecting the 
complexity of real-world settings, do not have a fixed solution, invite different solution approaches, stress 
collaboration among learners, and require learners to access a variety of information sources (pp. 32-33). Following 
Merrill's first-principle approach (2002), the tasks will be experienced as a series of (sub)tasks, beginning with tasks 
relating to gaining insight into the workplace problem or opportunity, moving into tasks that relate to planning a 
solution, then to tasks that relate to trying out the plans, and finally to tasks that relate to reflection. For each of these 
tasks, learners collaborate with each other in various ways, mediated by technology, and with scaffolding of support 
taking place, ideally moving from the course instructor or facilitator to the peers within the collaborative team and 
workplace supervisors. 

Subjects: All involved in the activity system, including not only the course instructor(s) and the learners 
directly enrolled in the course, but also the "learning partners" (Bianco & Collis, 2003) of the learners in their 
workplaces, particularly their supervisors and workplace colleagues who will become involved in the learning 
activities. 

Rules: In corporate settings, this is often the competency framework of the company as well as the formal 
and informal norms of behaviour. 

Community: The broader corporate context. In multinational corporations this becomes complicated as it 
involves not only a corporate culture but also the cultures of the individual workplaces. 

Division of Relationships: Workplace relationships affecting roles within collaborating teams and role 
distribution between learners, instructors, and the supervisors of learners. When multinational corporations are 
involved, there may be different expectations as to appropriate types of interactions among learners, instructors, and 
workplace supervisors related to different cultural backgrounds (Arya, Margaryan, & Collis, 2003). 

Instruments: This element has at least four components: a pedagogical foundation, learning resources, 
learning supports, and online tools and environments. 

Pedagogical foundation: In a recent synthesis of principles underlying instructional-design theories and 
models, Merrill (2002) has identified five key principles that he argues they have in common and thus represent a 
core basis for designing instruction. These are: 

"Learning is promoted when learners are engaged in solving real-world problems 

Learning is promoted when existing knowledge is activated as a foundation for new knowledge. 

Learning is promoted when new knowledge is demonstrated to the learner. 

Learning is promoted when new knowledge is applied by the learner. 

Learning is promoted when new knowledge is integrated into the learner's world." (Merrill, 2002, 
p. 44-45) 

These principles are "relationships that are always taken to be true under appropriate conditions, regardless 
of program or practice" (p. 43), and thus also appropriate for WBA-CSCL. They are "design oriented and 
prescriptive" . . . and thus "relate to creating learning environments and products" (p. 44) and thus provide a basis for a 
concrete realization of WBA-CSCL. Each of the five principles has corollaries that provide a translation from 
principle to prescription. These are summarized in Table 1. 

Table 1. Prescriptions for instructional design, summarized from Merrill, 2002, pp. 44-51 



Principles and corollaries 


Prescriptions 


Principle 1 : Problem setting 

Corollaries: Show task, task level, problem centered 


-Engage learners in real-world problems that are 
engaging and relevant 
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-Make clear the intended final behavior 

-Move in a progression from less-complex to more- 

complex tasks 


Principle 2: Activation 

Corollaries: Previous experience, new experience, 
structure 


-Apply past experiences to new problems, -Stimulate 
the development of mental models to bridge past 
experience with new problems 


Principle 3: Demonstration (Show me) 

Corollaries: Consistency, learner guidance, relevant 
media 


-Demonstrate information via specific examples or 
cases 

-Provide multiple representations 

-Guide the learner to focus on relevant aspects of the 

demonstration 


Principle 4: Application (Let me) 

Corollaries: Practice consistency, diminishing 
coaching, varied practice 


-Use new knowledge or skills to solve real-world 
problems 

-Scaffold the gradual withdrawal of support 
-Provide a variety of practice experiences 


Principle 5: Integration (Watch me) 
Corollaries: Reflection, integration 


-Synthesize and share 

-Create new insights and take into practice 



Learning resources: A variety of different types of resources should be available to support a 
WBA-CSCL pedagogy (Van der Veen, 1991; Oliver & Herrington, 2001). Because of the authentic 
workplace setting, some of the resources will be from the workplace itself, such as documents and artifacts 
relevant to the learning task. In the case of a multinational corporation, where learners involved in a 
collaborative task can come from local branches of the corporation that are highly diverse in culture and 
workplace climates, it is important that resources relate to the local settings of the learners as well as the 
general setting of their parent corporation. The need to anticipate cultural differences by providing a 
diversity of types of resources, as well as diversity in task expectations, then become a major issue (Arya, 
Margaryan, & Collis, 2003). 

Sources of learning resources can include: e-learning modules, manuals, help resources, and 
electronic performance support tools, digital capturing of in-house presentations or other artifacts from the 
workplace, resources available through the corporate knowledge management systems, such as discussion 
forums and best-practice repositories, professional reports located via the WWW or portals or conferences 
relevant to the domain of the problem. An important source of learning resources are the learners 
themselves, through the process of submitting the results of their various tasks into a shared electronic 
environment where they can be used and reused by others as learning resources. This "contribution-based 
pedagogy" is a key stimulus for learner-produced objects that can serve as learning objects for others 
(Collis & Moonen, 2001). 

Learning supports: Oliver and Herrington (2001) describe learning supports as "the processes and 
procedures by which learners are assisted in their learning activities, by which feedback and guidance is 
provided to them, and by which their involvement in the learning setting is encouraged and strengthened" 
(p. 55). This definition includes persons involved in learning support, including those directly involved (the 
course instructor or facilitators) and those involved without a role that explicitly indicates their support. 

The latter, in WBA-CSCL, include the peer-collaborators, the workplace supervisors or coaches who 
become involved in the activities by their nature of being a real workplace task, and others throughout the 
corporation whose experiences may be found within in-house discussion groups and repositories of best 
practice or who can be contacted by the learners in relation to their workplace tasks. 

Like Merrill (2002), Oliver and Herrington (2001) see collaborative learning and collaborative 
groups as learning support. They note factors that impact on the effectiveness of collaborative groups in 
terms of learning such as the extent to which the learners apportion tasks and carry out negotiation and 
synthesis. Van der Veen (2001) also identified a number of factors that constrain the effectiveness of 
collaborative problem solving, particularly when collaborators are at a distance from one another. He 
grouped these as operationalisation problems (for example, collaborators have problems organizing work 
between meetings, or accessing appropriate resources or problems arising to miscommunication); 
monitoring problems (such as facilitators lacking adequate insight into current learning activities); and 
planning problems (individual collaborators cannot keep to agreements about times for interactions and 
contributions). For each of these, support tools and strategies can help, particularly those available 
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electronically. As collaboration in the context of work-based activities will be regularly challenged by 
planning, operationalisation, and monitoring problems, support for collaboration must be considered within 
the WBA-CSCL framework. 

Online tools and environments: Electronic tools and environments are an integral component of 
CSCL, and therefore of the WBA-CSCL activity system. Networked tools and environments support the 
presentation of tasks and submission and feedback process associated with results of tasks. Shared 
environments provide the common medium for collaborative work, including for tasks related to planning, 
operationalisation, and monitoring. Tools and a shared environment are critical to a contribution-oriented 
approach to learning resources (Collis & Moonen, 2001). There are many categories of electronic tools for 
learning support for WBA-CSCL (Collis & Margaryan, 2003). These can include cognitive tools such as 
mind-mapping software or outlining software for problem or concept representation (Jonassen, 2003); 
calculators; modeling tools; tools to support brainstorming or note taking; and tools to support the 
scaffolding of learning (Winnips, 2001). As an example, Pramudya-Dharma (2003) has recently identified 
more than 70 open-source tools and systems available via the Web, which can support CSCL and work- 
based activities. The majority of the tools and systems in Pramudya-Dharma's compilation share aspects 
that can be said to scaffold learning, either explicitly, through devices such as prompts, or implicitly 
through the interactions (Jonassen, Peck, & Wilson, 1999; Winnips, 2001) that they make possible. 

The WBA-CSCL Lramework provides a framework for insights into learning as activity through "a form of 
sociocultural analysis that focuses on the activity system as the unit of analysis rather than the individual" (Jonassen, 
2002). However, the translation of such a framework into the design of CSCL learning events and environments 
requires a further step. "There is still an enormous amount of research needed to develop our understanding of how 
the material, social, and mental worlds interpenetrate in mediated activity. Activity Theory begins to lay out some of 
the dimensions of this task, but it is not yet clear how to apply the insights of Activity Theory (rather than merely 
critique) of Computer Supported Learning Environments" (Gifford & Enyedi, 1999). To move the WBA-CSCL 
Lramework from a descriptive system to a tool for realization, the "Instrument" component of WBA-CSCL must be 
translated into concrete terms. This involves operationalising a pedagogical model for WBA-CSCL. The next 
section will outline the steps of such a model and a methodology for practice. 

From Theoretical Framework to Pedagogical Steps 

In this section, a methodology to put the elements into operational steps is presented as guidelines for a 
WBA-CSCL pedagogical approach. 

The term "pedagogical model" is used in a variety of ways in the literature and practice but generally 
integrates aims, content, learning activities, and methodology within an underlying set of principles relating to 
learning or instructional design. (Grimmitt, 2003). Combining all the elements in the WBA-CSCL activity system 
discussed in the previous section, pedagogy to realize the WBA-CSCL in corporate practice can be expressed in a 
series of steps. The steps are divided into two parts, one for the design and preparation of a course including WBA- 
CSCL (Bianco, Collis, Cooke, & Margaryan, 2002; Merrill, 2002) and the other for facilitating the course in 
operation (Margaryan, Collis, & Cooke, 2003a,b). 

Design of a course involving WBA-CSCL: 

1 . Identify in general terms the business needs and workplace problems that will form the setting for the 
course. Lrom these, identify the general types of outcomes, the general types of work-based activities that 
are likely to be most related to the needs and problems, the forms of collaboration most likely to lead to the 
outcomes, and key concepts, skills, and knowledge related to the business needs and workplace problems. 

2. Decide on the high-level organization of the course: When might face-to-face sessions be desirable? How 
and when should the supervisors of the participants be involved? 

3. In more detail than Step 1, plan a series of tasks related to the overall work-based activity; for each task, 
identify what is expected, how the collaborating group should be working, when and how sharing with 
other groups will occur, and what learning resources and learning support are most likely to be relevant. 
Plan the contact and reporting moments within and among collaborating group. Plan the contact moments 
with the supervisors to engage them in the work-based activities. 

4. Select the electronic tools for CSCL support; design and set up the electronic environments that will 
support course processes. 

5. Advertise the course, make contacts with prospective learners and their supervisors explaining the WBA- 
CSCL aspects of the course, require all that wish to further continue to submit a "learning agreement" in 
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which the particular workplace needs and problems of the participants are indicated in detail and work- 
based activities that will be relevant in the workplace are identified. 

6. Retune the course planning, based on the knowledge of the specific workplace situations that will form the 
context for the WBA-CSCL that is obtained from the learning agreements. Refine descriptions of 
(sub)tasks, refine identification and selection of appropriate learning resources, refine plans for learning 
support. 

7. Elaborate the specific planning and support in the electronic course environment. Inform participants of the 
kick-off event for the course. 

WBA-CSCL in practice 

There is no single set of steps for how a course involving WBA-CSCL may be experienced in practice. 
Sometimes a face-to-face session will occur first, in order to orient the participants and cover some necessary 
preparation. Othertimes these aspects will occur on-line. Sometimes there will be no face-to-face setting for the 
course participants as a whole, other times there will be. The following however are generic steps, which should 
occur regardless of the approach chosen for course execution (adapted from Bianco, Collis, Cooke, & 
Margaryan, 2002; Collis & Moonen, 2001; Hendriks, 2003): 

1 . As the course is in operation, use the common electronic environment as support for each (sub)task. 
Participants can submit something that is subsequently (a) built upon for their own problem situation, (b) 
used by the other groups for peer sharing and interaction; and (c) is potentially available for reuse outside 
of the course context. 

2. For each submission, give feedback, from the course facilitators or via peers or other groups of experts. 
Make this feedback always available via the course Web environment. 

3. Check regularly with supervisors and engage them as "learning partners" (Bianco & Collis, 2003) 

4. Conclude the course with systematic reflection about the WBA-CSCL that has occurred and its relation to 
business needs as well as individual and group professional development. Integrate evaluation input from 
all major subjects (actors). 

5. Adapt the course accordingly for the following cycle 

This approach has been in realization in the past 18 months at the Shell Open University (now part of the 
Learning and Leadership Development Unit of Shell International Exploration and Production, LLD-SIEP) in 
collaboration with researchers from the University of Twente (for full reports see Arya, Margaryan, & Collis, 2003; 
Bianco & Collis, 2003; Bianco, Collis, Cooke, & Margaryan, 2002; Hendriks, 2003; Margaryan, Collis, & Cooke, 
2003a,b; Collis & Margaryan, 2003 ). As of September 2003, 96 courses have been redesigned to reflect the above 
methodology, although as yet not every course includes every step. In a detailed assessment of 36 of the courses 
(Collis, Margaryan, & Cooke, 2003a,b) instructions for 299 different work-based activities were found in the course 
Web environments. Evaluation has shown a consistently high valuation for the approach, by learners, instructors, 
and supervisors, although a common concern is that such an approach requires more time and effort than previous 
knowledge-transfer courses. Thus a start has been made, but research continues to be necessary. The experience that 
has been gathered at Shell has identified challenges that are the focus of on-going research. 

Challenges and Future Research 

The on-going research at the Shell EP Learning Centre confirms the expectation that moving to a new 
learning approach that integrates formal and informal learning around the framework of work-based activities and 
computer-supported collaborative learning is a complex process, but given a sound theoretical framework and a 
practical pedagogical model and course-redesign approach, scaling up into real practice can occur. Many issues 
remain challenging. These include: Organizational issues, time and workplace issues, issues relating to the 
involvement of the supervisor of the learner, and issues relating to the multinational setting. Each of these will be 
discussed briefly here. 

Organizational and social issues 

Major problems confronting the move to WBA-CSCL relate to the expectations in the workplace of what 
constitutes "learning". Learning is still seen, and accredited, around time spent "at courses". The concept of learning 
as going away to a course is strongly embedded not only in the expectations of all in the organization but also in the 
procedures used to describe and fund formal learning (how many people x how many hours in the classroom). 
Informal learning, or learning that goes on while still doing one's job, is not yet part of the formal system of 
recognizing learning in terms of promotions and career ladders although at Shell EP is being accepted as part of the 
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new learning strategy adopted by the company in 2003. However, despite a statement of corporate strategy, support 
in terms of released time and learning facilitation is not yet in place when learning takes place within the context of 
real work in the learning setting. This places WBA-CSCL in a fringe or periphery position in terms of support and 
internal recognition. 

Time and workplace issues 

The strength of WBA-CSCL is that it is grounded in real workplace tasks. But this also contributes to the 
problem of how to build in the time needed for collaborative learning above simply "doing the job". The result is 
often that the learner has to fit in any collaborative-learning activities, particularly those relating to sharing, peer 
interaction and reflection, into his own time outside of the workplace. While many may be motivated enough by the 
benefits of learning from colleagues and being guided through elements such as reflection, many more will just not 
have the time to do more than their regular job. The close boundary between WBA-CSCL and learning is not only 
its strength but also its vulnerability in practice. 

Issues relatiug to the iuvolvemeut of the supervisor 

The supervisor of the learner involved in WBA-CSCL must be supportive and should be more than only 
supportive, but also involved in the learning processes (Bianco & Collis, 2003). In practice, this step is difficult to 
achieve. Supervisors see "learning" as "going away to courses" and do not see that they have any further 
involvement. Many different strategies are being attempted, but without building support for WBA-CSCL into their 
job descriptions, the effort to effectively involve supervisors is an uphill battle. 

Issues relatiug to the multiuatioual settiug 

As the research has already shown, there are many complexities involved with striving for effective 
collaborative learning when the collaborators are distributed among different parts of the world and represent 
different cultural backgrounds. One approach is that of assuming homogeneity: in a multinational company, the 
course instructor might expect that all the learners will benefit from being treated as fitting a single "company 
culture". However, differences among people will still remain, and a more-productive approach will be to leverage 
cultural differences. How to realize this and make it manageable within a pedagogical model is a major challenge. 

Thus, the WBA-CSCL Framework and its pedagogical realization appear to help in structuring the 
introduction of new forms of learning for professionals in multinational corporations. However, much research 
needs to be done, and much needs to change in the social climate of the workplace before the potential of the 
approach will be realized in daily practice. 
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Practices and Academic Preparation 
of Instructional Designers 

Suzy Cox 

Brigham Young University 

The field of Instructional Design is relatively young-just over 50 years old according to some researchers. 
As is common in immature fields. Instructional Design has struggled to define itself. Definitions and theory were 
first derived from other disciplines, including Psychology, Computer Science, Statistics, and Education. While these 
disciplines continue to influence Instructional Design, the field is beginning to develop its own sense of purpose and 
scope. 

Over the 50 years of its existence, many researchers have attempted to define Instructional Design. These 
definitions have been useful in guiding the fledgling field, but many are not informed by a careful examination of 
practice. Most have been very general definitions ranging from descriptions of practitioners’ positions with respect 
to the use of hardware and software to the intangible theoretical bases for the field. In fact, even deciding on a name 
for the field has been a difficult challenge. I used the title of Instructional Design in this study because I examined 
the practice of instructional designers and the instructional design process. According to Reiser, the practice of 
instructional design is composed of two parts: first, “the use of media for instructional purposes,” and second, “the 
use of systematic instructional design procedures” (2001). Seels and Richey define Instructional [Design] as “the 
theory and practice of design, development, utilization, management and evaluation of processes and resources for 
learning” (1994). While these definitions are succinct, they do not reveal a full understanding of an instructional 
designer’s responsibilities in the workplace. 

As the field has matured and evolved, more attempts have been made to allow the theory and the practice 
of the field to interact with and inform one another. This is evidenced by the competencies for instructional 
designers (Richey, 2001) and the standards for academic programs (Earle, 2000) that have been developed as well as 
recent studies examining job descriptions in Instructional Design. Eor example, in an attempt to identify specific 
skills needed by designers, a study was conducted to determine the instructional design competencies that are most 
often requested by instructional design employers (Byun, 2000). This five-year study described 93 competencies 
that were then clustered into 33 relatively general skill categories (i.e., management, leadership, instructional design 
and development, etc.). While the information provided in this study is certainly useful and fulfills its purpose of 
informing academic departments of the most sought-after skills, it does not reveal what is actually being practiced 
once an individual is hired. 

Theories in the field have also not generally been derived from professional practice (Reigeluth, 1999). 
While academic researchers may possess a vision of Instructional Design as a discipline, practitioners are 
comparatively unaware and uninformed of the purpose and direction of their field (Gibbons, 2000). Perhaps the best 
way to define the functions and parameters of a field is to observe the daily responsibilities of its practitioners 
(Stolovitch, 1995). Such studies have been conducted in many fields, including accounting (Pearson, 1991), medical 
education (Morrison, 2000), and teacher education (Adelman, 1998; Stark, 1985). Studies of this nature reveal the 
specific tasks and responsibilities performed by practitioners in each field. However, similar studies have not been 
conducted in Instructional Design in the recent past. While some research was conducted related to this topic in the 
early 1990’s (G. Rowland, 1992; Wedman, 1993; Winer, 1995), these pre-Internet studies do not paint an accurate 
picture of current practice and many were conducted using a very small sample of designers. In recent years, studies 
in this field have relied on practitioners’ and employers’ perceptions of important skills and knowledge and not on 
actual reports by designers themselves of their daily practice. There seems to be an incongruity between academia 
and practice as institutions continue to teach theories, models, and concepts that may or may not be relevant in the 
professional world. While job descriptions and competencies may list these theories, models, and concepts as 
essential in the workplace, the practitioners themselves have not yet confirmed that such theories and models 
actually guide their practice. 

Questions raised by an initial study I conducted in 2002 (Cox, 2003) clarified the need for further study of 
the instructional design profession. What, exactly, do instructional designers do? Which tasks are required of them 
regularly and which do they seldom engage in? How much of their time is spent in meetings and other 
organizational tasks and how much is spent actually designing instruction? 

Determining how designers spend their time yields another very important question: are academic 
programs adequately teaching designers the skills they most often use? Job descriptions in the field typically include 
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requests for applicants to possess advanced academic degrees (Byun, 2000). Yet it has not yet been determined if the 
designers themselves feel that Instructional Design graduate programs are preparing them for the job market. 
Curriculum may emphasize learning theories or ISD models but lack programming courses or other technical 
curriculum that is useful in practice. Because of this, graduates may leave their institutions feeling unsure of their 
skills or prospects for employment. 

In this study I attempted to identify those tasks that professional instructional designers perform every day 
and the perceived value of advanced degree programs in preparing them to perform those tasks. This close 
examination of the practice of the discipline not only begins to reveal the scope of Instructional Design as a 
profession but will also inform those who offer academic programs as they work to prepare students for the job 
market. The study also reveals subtle differences in responsibilities based on workplace and experience. Finally, it 
identifies areas in which academic preparation may be lacking and suggests ways in which programs might improve 
or change their curriculum to meet practitioner needs. This information will be framed in the context of the recently 
adopted standards for programs in educational communications and instructional technologies (ECIT). 

Research Questions 

This study centers on the following research questions: 

1 . How do instructional designers spend their professional time? 

2. How well do instructional designers feel their graduate programs prepared them for their assigned 
responsibilities? 

I was also interested in studying the differences in how designers spend their time based on workplace, 
level of experience, and level of education and how the highest academic degree earned might be associated with 
differences in feelings of preparation. 

Method 

Sample 

The first step in conducting this survey was to define the population to be examined. Because one of the 
major research questions sought to identify the level of preparation of practitioners who have graduated from formal 
Instructional Design programs, only alumni of such programs were contacted for participation. Acquiring the alumni 
lists proved to be the greatest challenge in completing the study as each university had a different policy with regard 
to their distribution. While some were allowed to release their lists directly to me, others offered to send the survey 
invitation themselves in order to protect their graduates. Some universities were unable to participate for various 
reasons (i.e., too busy, restricted access to alumni lists, etc.). This limited the scope of the study to the following 
institutions: Brigham Young University, Utah State University, University of Indiana, University of Georgia, 
Syracuse University, Florida State University, Pennsylvania State University, and San Diego State University. While 
this is not a long list, these institutions are among the most widely recognized programs in the field. 

It should be noted here that the group that participated in this study is a subpopulation of the whole. As I 
stated earlier, the field of instructional design is in flux and searching for direction. By focusing on some of the 
better-known graduate programs in the field, I hoped to be able to paint a picture of what the leading programs are 
accomplishing and where we might be headed. For this reason, the results of the study may not be representative of 
what is happening at all institutions or with all instructional designers. 

Further, I decided to limit the sample to full-time practicing instructional designers in order to make the 
scope of the study more manageable. 

Instrumentation 

Because technical proficiency is considered to be a mandatory skill in Instructional Design (Dillon, 2001), 
it was reasonable to assume that the target population would have access to and familiarity with the required 
technology to receive and complete an online survey. A link to the survey, along with a brief description of its 
purpose, was sent to each alumnus via email. Individuals who did not provide an email address to their institutions 
were not contacted for participation. I was able to contact alumni from Brigham Young University, Utah State 
University, and Florida State University directly, which enabled me to obtain an accurate response rate for these 
institutions only. All other alumni were contacted through a generic message sent by a representative at their 
universities. 

After passing through a password page, consent form, and demographic questions, prospective participants 
were asked a qualifying question regarding current employment status. Only those who said they were working full- 
time as instructional designers could participate in the study. This measure was intended to insure that the survey 
data would not be corrupted by graduates who have pursued careers in which their primary function is not 



131 




instructional design. This qualifying question also provided a more accurate survey response rate by allowing all 
alumni to begin the survey. 

The survey itself began with some basic demographic questions including job title, employer type, 
academic degrees earned, and years of experience. Participants were then asked to indicate the percentage of their 
professional time spent in instructional design tasks and organizational tasks. Next, they were asked to indicate what 
percentage of their instructional design time was spent in various capacities including analysis, design, development, 
implementation, and evaluation (ADDIE). While there are many existing ISD models that might have been used to 
organize the tasks, the ADDIE model stands as the most widely accepted and is the basis for many other models 
(Rosenberg, 1999). It is noted that some consider the ADDIE model to be too restrictive and a few participants 
indicated that it was difficult to fit their responsibilities into those limited categories. Because of the very small 
number of comments in this area, I believe that the broad nature of the categories and the further definition of tasks 
in subsequent questions alleviated much of the discomfort over the use of the ADDIE model. 

Organizational time was divided into responsibilities such as project management, personnel supervision, 
marketing, research, and professional development. Participants were then presented with a more detailed list of 
both design and organizational tasks and asked to identify how often they perform each task in their job and how 
well they feel their graduate programs prepared them for those duties. Both frequency and preparedness were 
measured on a rating scale in order to reduce the cognitive load on participants. Pilot tests revealed that attempting 
to assign percentages to each of the individual tasks was overwhelming for respondents. These detailed tasks were 
derived from a number of sources including Seels & Richey’s {\99A)Instructional Technology: Definition and 
Domains of the Field, the results of the initial study that I conducted, and interviews and pilot tests with practicing 
designers. The final section of the survey consisted of affective questions regarding the participants’ academic 
preparation and the adequacy of their training. 

Analysis 

I used t-tests to compare the means of different groups for those questions in which participants were asked 
to assign a percentage of their time to different tasks. I created cross-tabulation tables and calculated chi-square tests 
on those questions that involved rating a variable on a provided scale. Cross-tabulation of variables was performed 
to determine if different groupings had an impact on how often various tasks were performed, how well prepared 
participants felt to perform those tasks, and how confident respondents felt in their ability to perform their 
responsibilities overall. Chi-square tests and t-tests were calculated for the following groupings: workplace, level of 
experience, highest degree earned, and school attended. 

Results 



Response Rate 

As mentioned earlier, only alumni of Brigham Young University, Utah State University, and Elorida State 
University were contacted directly. All other participants were contacted by a representative of the university from 
which they graduated through a generic email. Eor this reason, an accurate response rate can be determined only for 
BYU, USU, and ESU. The response rate for these institutions, after accounting for undeliverable mailings, was 63%. 
This compares favorably with rates typically seen in traditional surveys (Heberlein & Baumgartner, 1978, as cited in 
Kiesler, 1986). 

Demographic Information 

Of the total number of respondents, 56% indicated that they are currently employed full-time in an 
instructional design capacity. As mentioned above, this question was open to some interpretation and, therefore, the 
following results may include data from some individuals who are not necessarily instructional designers according 
to the definition given earlier. Some participants indicated through comments in the survey that they were employed 
in educational technology or administrative positions, but responded positively to the screening question. As it is 
impossible to determine if other participants did the same, the results of these participants were included in the final 
analysis. We are only able to conjecture how the remaining 44% are employed. This ambiguity suggests that, in 
future studies, participants be allowed to select a position title rather than answer a yes/no question regarding 
employment to gain a more descriptive picture of this demographic. An earlier, informal study that I conducted 
suggests that those not employed as instructional designers are faculty, administrators, instructional developers, 
project managers, teachers/trainers, or students. Others may have left full-time employment for retirement, part-time 
employment, or family or other responsibilities. 
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Among those currently employed in instructional design, the vast majority are employed in higher 
education (38%) or a large business setting (30%). “Large business” is defined here as a business with over 500 
employees (Small Business Administration website). Other categories include smaller businesses (9%), self- 
employment (7%), K-12 schools (5%) and the government and military (5%). Because there were so few 
respondents in some of the employment categories, they were collapsed into four groups: Academics, Government, 
Small Business, and Large Business. 

Survey participants ranged in experience from 0-40 years. This range was collapsed into the following 
categories: 0-1 years, 2-5 years, 6-10 years, 1 1-20 years, and more than 20 years of experience. The average amount 
of experience for all participants was nine years. One preceding study identified experts as those with at least seven 
years of experience (G. Rowland, 1992). Just over half of our sample has less than seven years of experience in 
instructional design. This data indicates a relatively young work force. 

Research Question 1: How do instructional designers spend their professional time? 

How do instructional designers spend their time? Respondents indicated that, on average, 53% of their 
professional time is spent in organizational tasks while 47% was spent performing instructional design tasks. These 
numbers may result from my definition of the task categories. Organizational tasks included project management, 
supervising personnel, professional meetings, academic research, marketing/sales, and professional development. 
Some may argue that any of these categories might be considered part of the instructional design process. The 
decision was made, however, to limit the instructional design tasks to those encompassed by the ADDIE model- 
analysis, design, development, implementation, and evaluation. Participants were then asked to divide their 
instructional design and organizational time into more specific task categories. 

When asked to divide their instructional design time by task category (analysis, design, development, 
implementation, or evaluation), development received 29% of the time distribution with design at 21%. These were 
followed by analysis, implementation, and evaluation (See Figure 1). 

Instructional Design Tasks 



Implementation 

17% 



Evaluation 




Figure 1. Distribution of instructional design time by category. 



Each task category was divided into individual tasks that respondents rated according to the frequency with 
which they performed the task. The instructional design task most often performed involved writing and editing 
instructional materials (mean = 3.03 out of 4.0). The task least often performed was programming at 2.0. See Figure 
2 . 
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Instructional Design Task Frequency 
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Figure 2. Average frequency ratings for all instructional design tasks. 



When asked to divide their organizational time by task category (project management, supervising 
personnel, professional meetings, academic research, marketing/sales, or professional development), project 
management and professional meetings consumed the majority of respondents’ time; 28% and 24%, respectively. 
These were followed by supervising personnel, academic research, marketing/sales, and professional development 
(See Figure 3). 
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Organizational Tasks 



Professional Development 
9% 




Figure 3. Distribution of organizational time by category. 



As was done with instructional design, the organizational task categories were divided into individual tasks. 
The organizational task most often performed involved reporting to and interacting with supervisors-a task in the 
“professional meetings” category. In fact, all tasks within this category were rated above a 3.0 (often). The least 
performed task was making marketing presentations at conferences (1.73 on the 4-point scale). See Figure 4. 

I was also interested to see how various factors influence the ways in which designers spend their time. 
Results were analyzed based on workplace, level of experience, and level of education. Using t-tests, no significant 
differences were found in the overall division of participants’ time between instructional design and organizational 
tasks, but chi-square tests reveal that the frequency with which specific functions are performed does seem to be 
associated with these factors. 

Designers who work for small businesses tend to spend significantly less time designing assessments than 
any other group. While budgeting tasks are not frequently performed by any group, instructional designers in large 
businesses spend much less time performing this function than any other group — over 50% indicated that they never 
create budgets. Not surprisingly, those in academic positions (K-12 or College/University) engage in research 
activities including designing, conducting, and publishing research significantly more often than their business and 
government counterparts. This group also attends professional conferences more often than all other groups. 

Level of experience is also associated with the frequency with which designers engage in individual tasks 
such as selecting instructional strategies and programming. Senior designers participate more often in the selection 
of strategies while newer designers are more involved in programming tasks. 

Respondents with doctoral degrees tend to spend more time researching and providing rationale for design 
while Master’s degree holders participate more often in storyboarding. Those with doctoral degrees also spend more 
time in evaluation tasks, particularly formative evaluation. Master’s program graduates supervise personnel more 
often and meet more often with cross-functional co-workers (e.g., technical, quality assurance, marketing, or sales). 
Not surprisingly, those with doctoral degrees participate more often in tasks that involve research, including 
designing research, reading scholarly publications, and writing articles. They also attend professional conferences 
more frequently. 
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Organizational Task Frequency 
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Figure 4. Average frequency ratings for all organizational tasks. 

Research Question 2: How well do instructional designers feel their graduate programs prepared them for their 
assigned responsibilities? 

Survey respondents indicated that they felt fairly well prepared by their graduate studies for their 
instructional design duties, with an average score of just over 3.0 on a 4-point scale (1 = very poorly, 2 = poorly, 3 
fairly well, or 4 = extremely well prepared). As shown in Figure 5, they felt most prepared for duties involving 
instructional strategy design (3.63) and least prepared for programming tasks (2.34). 
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Instructional Design Task Preparedness 




Figure 5. Average preparedness ratings for all instructional design tasks. 

Survey participants indicated that they felt slightly less prepared overall to perform their organizational 
duties. The average preparedness rating for organizational tasks was 2.6 on our 4-point scale. As shown in Figure 6, 
the task for which they felt most prepared was academic research involving reading journals and trade books for 
research purposes (3.47). Participants felt least prepared to recruit, interview, and hire staff (1.82). In fact, three of 
the four tasks within the category of “supervising personnel” received mean ratings below a 2.0 on this scale. 

There is a weak positive correlation between task frequency and task preparedness (r = 0.5 1 for 
instructional design tasks, r = 0.40 for organizational tasks), suggesting that, to some degree, graduate programs are 
emphasizing the most often performed tasks. For example, while respondents indicated that programming is the 
instructional design task for which they feel least prepared, they also indicated that it is the least often performed 
design task. At the other end, participants chose instructional strategy design as the task for which they feel best 
prepared. This task is also the second most frequently performed. 

I was interested to see how level of education influenced feelings of preparedness in the survey 
respondents. In this survey, those with Master’s degrees feel better prepared to conduct a needs analysis, while 
doctoral graduates are more comfortable in all of the research-related tasks. Doctoral degree holders are also 
somewhat better prepared to take professional development classes. 



For what responsibilities do they feel best prepared? 

When asked to select the one task for which they felt best prepared by their graduate studies in the 
instructional design tasks category, 29% of participants chose instructional strategy design. An additional 17% 
selected needs analysis and 15% picked researching and providing theoretical rational for their design as the task for 
which they were best prepared. The remaining responses were distributed among a variety of choices. It is 
interesting to note that no one chose performing competitive product reviews, writing implementation materials, or 
piloting prototypes as their best task. 

In the organizational tasks category, 18% of the participants said that they were best prepared to design and 
conduct research. An additional 17% selected meeting with project teams. The remaining responses were distributed 
among a variety of choices. None of the respondents indicated that they were best prepared to conduct performance 
reviews or make marketing presentations at conferences. 



137 



Organizational Task Preparedness 
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Figure 6. Average preparedness ratings for all organizational tasks. 

For what responsibilities do they feel least prepared? 

In the instructional design tasks category, 24% of participants chose programming as the task for which 
they were least prepared, 21% selected competitive product reviews, and 10% picked media production. No one 
indicated that they felt least prepared to write or edit instructional materials, plan implementation strategies and 
training, or plan and conduct formative evaluations. 

Budgeting was selected most often as the organizational task for which respondents were least prepared 
(24%). Additionally, 19% selected tasks involving recruiting and hiring staff, 11% chose working with contract 
personnel, and 10% picked designing marketing strategies. None of the survey respondents chose meeting with the 
project team, planning those team meetings, or reading journals for research purposes as the task for which they felt 
least prepared. 



Discussion and Conclusions 

How do instructional designers spend their time? Those in this study spend a slight majority of their time 
participating in organizational activities. Their instructional design time is spent largely in designing and developing 
instruction. While, on average, they engage in all of the steps in the ADDIE model of instructional design, none of 
the phases of the model are performed very often by the group. They most frequently participate in instructional 
design tasks that involve writing and editing instructional materials and instructional strategy design. The most 
frequently performed organizational tasks involve meetings and project management. 

The participants in this study feel fairly well prepared to engage in the tasks required of them, but there is 
room for improvement. One question in the survey asked participants what they would change about their graduate 
studies. Three comments were prevalent; (a) more business/organizational training, (b) more technical training, and 
(c) more real-world application in the form of teamwork, consulting, internships, and existing product reviews. 
Interestingly, these comments are directly aligned with changes suggested by at least one earlier researcher (Quinn, 
1994). More than one respondent suggested that instructors attempt to create a level of unification between classes 
so that students are able to make connections and supervise projects through the entire process. Another participant 
made a very frank comment that, I believe, holds great potential for program improvement. He said: 
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"For heaven's sake, give us some decent project management classes. Help to learn to 
make budgets and follow them. Teach us to work in teams. Help us to estimate costs. On 
the ID side, we need more projects that are in a real setting as part of our class. We also 
need more projects that we step through the entire design process on. It would be great 
if we could set up a situation where local community entities with instructional needs 
could get a team of students to meet their needs. You can't substitute for real experience 
in a real setting. Also, bring in the practitioners, not just for lectures but to critique 
project plans or project results. I could have learned a lot from some professional eyes 
looking at my student work." 

In short, practicing designers want their academic programs to more closely mirror the 
experience of being a designer in the real world. 

Again, the participants in this survey were all graduates of some of the more prominent academic programs 
in the field, and all have received an advanced degree in instructional design. While this distinction may cause the 
results of this study to be less representative of the larger population of instructional designers and institutions, it is 
reasonable to assume that these programs will serve as models for newer departments in the field. For this reason, I 
believe that a study of these institutions and their graduates helps to paint a picture of the present and future of our 
field. 

While the graduate programs in this study do a fairly good job of preparing instructional designers for 
employment, there seem to be some areas in which improvement could be made. First, an increased emphasis in 
business skills may be valuable. Second, administrators of Master’s degree programs in particular may benefit their 
students by emphasizing proficiency in development and production skills including programming, storyboarding, 
prototyping, etc. Third, administrators should put forth more effort in informing students of the differences between 
job environments so that graduates are better able to make employment decisions. Potential students may also want 
to examine the emphasis of each program they are considering in order to select the one that is most in line with 
their career goals. Another consideration may be the efficacy of undergraduate programs in the field. Approximately 
67% of those surveyed felt that an undergraduate degree would adequately prepare individuals for an instructional 
design position. 

Finally, degree programs must offer more than in-class instruction and research projects. An interesting 
comment made by two respondents involved the nature of their graduate experience. They answered the survey 
questions according to their graduate coursework, as instructed, but mentioned that other experiences, including 
internships and faculty mentoring, prepared them for future employment much better than their coursework did. 
Other researchers (Ertmer, 1995; Quinn, 1994; Rowland, 1994; Tripp, 1994) agree that providing instructional 
design students with practical experience in the form of internships, apprenticeships, class projects, or design studios 
may be the best way to prepare them for the workforce. 

The above recommendations are not new. Though similar suggestions for improvement were made nearly a 
decade ago (Quinn, 1994), it would appear that none of them have been widely implemented. It is time that program 
administrators listen to their graduates and the practitioners of the discipline and make the necessary changes so that 
instructional design alumni can be better prepared to fulfill their responsibilities. 

Suggestions for future research 

This study has introduced a number of questions that I feel should be addressed in future research. The first 
involves the various capacities in which graduates of instructional design programs are employed. My second 
suggestion is a more in-depth examination of designer practice including case studies of individual designers 
employed by various institutions. A third potential research topic is the curricular offerings of academic programs in 
the field and how they compare to the ECIT standards. Einally, I believe that an examination of the existence and 
composition of instructional design teams would be beneficial to the field. 
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Abstract 

Network based education systems are important in this technologically advanced era. ALEX, one such 
system, is proposed in this paper as a web portal for Alabama teachers. The ALEX system includes lesson plans, 
web resources, communication tools, and a wealth of other educational materials for teachers. HTML, PHP, PERL, 
and MySQL were used to develop this system; appropriately, it is deployed on the Internet making it available to all 
Alabama educators regardless of geographic location or economic status. 

Introduction 

The Alabama Learning Exchange (ALEX), http;//www.alex.state.al.us, is a network based education 
system that was designed to help improve education in the state of Alabama. The ALEX system provides one 
location for Alabama K-12 educators to find information that will help improve their effectiveness in the classroom. 
It includes courses of study content standards, lesson plans, web resources, listservs, a bulletin board, and other 
educational resources. 

The Alabama Department of Education, http://www.alsde.edu. Office of Technology Initiatives (OTI) 
wanted an education web portal. They asked the Alabama Supercomputer Authority (ASA), http://www.asc.edu, to 
develop, host, and maintain it. ASA is a state agency that provides a supercomputer for the state. The 
supercomputer is used mostly by higher education for research. ASA also runs the Alabama Research and 
Education Network (AREN), which provides Internet access for Alabama public K-12 school systems, libraries, 
postsecondary colleges, four year colleges, and universities. OTI was familiar with ASA’s work because of the 
technical support they provide to the Alabama Virtual Library (AVL), http://www.avl.lib.al.us, and its role as the 
Internet Service Provider for Alabama’s educational institutions. 

At the time, there was a significant need for a website that would provide teachers with a way to access 
teaching guidelines, search them, and find lesson plans that would assist them in the teaching of those guidelines. 
ALEX provides one central location for educators to find teaching resources and collaborate with statewide 
counterparts. Before the deployment of ALEX, most teachers shared one book of content standards for each course 
of study. Content standards are guidelines that teachers are required to use when teaching students. There are 
content standards for each course of study or subject. Because of this new education portal, accessing the content 
standards is more convenient for teachers. 



ALEX System 

Erom a development standpoint, ALEX is divided into seven sections as illustrated in Eigure 3.1. They are 
teacher-created lesson plans, personal workspace, courses of study content standards, web resources, teacher-to- 
teacher, teacher zone, and MarcoPolo lesson plans. 
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Figure 3.1 ALEX System Architecture 



Additional sections have been added to make the site user friendly and easy to use. These sections include 
the search, help, suggestion box, and site map sections. 

System Configuration 

The system, as illustrated in Figure 3.3, consists of two Linux servers. One is the web server and the other 
is the database server. The web server has Red Hat Linux release 7.2 installed, and the kernel version is 2.4.7-10. It 
runs Apache software version 1.3.22. It has one 600 MHz processor with approximately 20GB of disk space and 
256MB of RAM. The scripting languages used in the development were PHP and Programming Extracting 
Reporting Language (PERL) which both reside on the web server. The database server has Red Hat Linux release 
7.3 installed with kernel version 2.4.18-19.7.xsmp. It has two 2.2GHz processors with approximately 758GB of 
disk space and 1GB of RAM. The database server runs MySQL version 3.23.56. 




Figure 3.3 ALEX Servers 



Courses of Study Content Standards 

Alabama has content standards, guidelines, for each course of study or subject. The tblStandards database 
table was created to hold content standards for all subjects. The table was designed to make sure that data could be 
displayed very similarly to the way it was displayed in the course of study content standard books distributed by the 
Alabama Department of Education to K-12 teachers each year. These content standards can be browsed and 
searched by subject and grade. A keyword can also be specified in a search. Results are presented in two formats. 
One format is very similar to the way standards appear in the book. The other format is a printer friendly version. 
Both versions include the course (subject), year, grade(s), topic (within the subject area), and links to related web 
resources and lesson plans for each content standard. 
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Personal Workspace 

ALEX grants a personal workspace to users that register to create an account. An online form is provided 
for registering, users are asked to enter their name, a user name, and an email address. When the user clicks submit, 
this information is stored in the user database table. To complete the registration process, users must respond to a 
system generated email message from the email account they specified when registering. Within this email message 
is a link that the user must visit. The link includes a hash and an email parameter. These 2 parameters are compared 
with corresponding values that were stored in the user database table when the online registration form was 
submitted. Once confirmed, the user is asked to enter their affiliation information before they are allowed to use 
their personal workspace. Most users are affiliated with schools or other educational organizations. 

The information presented in the personal workspace is dependent on the type of user that is logged in. If 
the user is a general user with no administrative or approval privileges, then they can create, modify, submit, and 
delete lesson plans and web resources. Users can also add and delete bookmarks to approved teacher-created and 
MarcoPolo lesson plans of interest, view administrative messages, and participate in groups. If the user is an 
administrator they can view additional items such as lesson plans, web resources, and variations awaiting approval, 
as well as other administrative reports. 



Teacher- Created Lesson Plans 

Teachers who register can use their personal workspace to create lesson plans. The “Create Lesson Plan” 
link appears on the personal workspace for all user types. A lesson plan interface is composed of three forms. The 
first form is where the user fills in descriptive information about the lesson plan such as title, materials needed, and 
duration of the lesson. Most of this information is required and JavaScript is used to verify that the user populates 
the required fields. From this initial interface the user has two options. They can associate this lesson plan with 
content standards or create steps. The lesson plan cannot be submitted for approval until it has been associated with 
content standards. First, to associate or align the lesson plan with content standards, the user is presented with a 
search screen. The user must select the course of study and grade that correspond with the lesson plan at hand. The 
user is then presented with content standards matching their search criteria. After aligning the plan by clicking the 
“Save Associated Standards” button, the user is returned to the initial interface where the aligned content standards 
now appear. From here, the user can opt to create steps. These steps make up the procedure for the lesson plan. A 
plan can have as many steps as necessary. Each step includes information like URL and duration for that particular 
step. Users can also associate attachments to the plan by uploading them via this interface. Once created, users can 
return to their personal workspace to modify, delete, or submit their lesson plans for approval. 

Approval Process 

Once submitted by the user, the lesson plan must go through an approval process before it can be included 
in the database of teacher-created lesson plans searchable by the public. Users that have been designated as 
approvers can view a listing of submitted lesson plans on their personal workspace in the “Waiting Approval” 
section. Approvers select the lesson plan they wish to review. Then they are presented with an approver’s screen, 
where they can view the lesson plan or check it out for a grammatical evaluation and/or a content evaluation. 
Approver’s can choose to check out a lesson plan for grammatical evaluation or for content evaluation. They can 
also choose to check out a lesson plan for both grammar and content. This decision is based on their area(s) of 
expertise. Approver’s are asked to check out plans so that other approver’s know that the plan is already being 
evaluated. Once a plan has been checked out once, it has to be un-checked before another approver is allowed to 
check it out. Plans are unchecked on the same screen that they are checked out on. This eliminates multiple 
approvers evaluating the same plans for approval purposes. Only after an approver has checked out a lesson plan 
can they reject or approve it based on grammar and/or content. Both grammar and content must be approved before 
the lesson plan is considered approved. If the approver chooses to reject the plan for either grammar or content they 
can provide comments to the submitter in the form of a message, which will appear on the user’s personal 
workspace. This helps the submitter determine why the plan was rejected and make suggested changes before 
submitting the lesson plan again. The approver can also edit the plan themselves as needed. If the plan is approved, 
the plan submitter is sent a message informing them that their plan has been approved. That plan is now included in 
the publicly searchable database of teacher-created lesson plans. 

Group Functionality 

If users have been designated as group leaders they have additional group options to choose from. They 
can create and rename groups that they are administering; they can add, view, and delete group members. They can 
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also approve or reject lesson plans submitted to them by group members. Group members can submit their lesson 
plans to the group leader, rather than directly to the approval process. If the group leader chooses to reject the plan 
he/she can provide comments to the plan’s creator, which will appear in the message section of the creator’s 
personal workspace. If the group leader chooses to approve the lesson, it is submitted to the approval process. As 
described earlier it must be approved before it can be included in the publicly searchable database of teacher-created 
lesson plans. Group members may choose to leave the group at anytime. 

Ranking 

Portal users can anonymously rank approved teacher-created lesson plans. They are asked to rate the 
usefulness, content, and creativity of the lesson plan based on a scale from 1 to 5. Users may also provide 
comments along with these ratings. The results are accessible on the personal workspace of administrative users 
under “Ranking Results”. The average ratings for usefulness, content, and creativity are shown along with any 
comments that may have been submitted along with the ratings. The date the ratings were submitted and the IP 
address of the submitter are also included. This precautionary information was added, so administrators would 
know if a user was trying to skew the results by ranking a lesson plan several times. 

Variations 

Users can suggest modifications to approved teacher-created lesson plans. An interface is provided where 
they can suggest that certain areas of a lesson plan be modified to expand its usability and/or to improve its 
effectiveness. Attachments may also be uploaded to support the suggested modifications. These changes must then 
be approved. If the lesson plan is rejected the approver can supply comments to the submitter explaining why. If 
the modifications are accepted they are appended to the existing lesson plan. From that point on, the variation will 
always appear with that lesson plan. 



MarcoPolo Lesson Plans 

The MarcoPolo Education Foundation, http://www.marcopolo-education.org, has seven partners that 
provide MarcoPolo lesson plans. The John F. Kennedy Center for the Performing Arts, http://artsedge.kennedy- 
center.org, provides lesson plans dealing with the arts; the National Council on Economic Education, 
http://www.econedlink.org, provides economics related lesson plans; the National Endowment for the Humanities, 
http://edsitement.neh.gov, provides humanities types of lesson plans; the National Council of Teachers of 
Mathematics, http://illuminations.nctm.org, contributes mathematical lesson plans; the International Reading 
Association and The National Council of Teachers of English, http://www.readwritethink.org, contribute reading 
and language arts lesson plans; the American Association for the Advancement of Science, 
http://www.sciencenetlinks.com, provides science lesson plans, and the National Geographic Society, 
http://www.nationalgeographic.com/xpeditions, provides geography lesson plans. These MarcoPolo lesson plans 
have been incorporated into the ALEX network based education system. Website visitors can search for lesson 
plans by specifying subject(s), grade(s), and specific topic(s) within the chosen subject area(s). 

Alignment 

MarcoPolo lesson plans are aligned to the Alabama courses of study content standards. Users who have 
been designated MarcoPolo aligners can search for MarcoPolo lessons to review. To review lessons, aligners are 
presented with the MarcoPolo lesson plan search interface. Aligners can choose the subject, grade, and topic to 
search for relevant MarcoPolo lesson plans to align. A list of MarcoPolo lesson plans is retrieved. This listing 
includes links to MarcoPolo lesson plan(s) that match the search criteria and a list of content standards that the plans 
are already aligned to. Once a lesson plan has been reviewed aligners can align it to corresponding content 
standards. To do this aligners are presented with another very similar search interface. In fact, it is the same 
interface that is used to search for course of study content standards described earlier. Aligners select course of 
study, grade, and topic; they also have the option of entering a keyword. The resulting list of MarcoPolo lesson 
plans is then aligned to content standard(s). 



Web Resources 

Users can search for web resources that have been correlated to Alabama content standards and view pre- 
selected web resources that have been categorized by audience and type. Registered users can submit web resources 
to be included in this list of categorized resources. Furthermore, submitted resources are correlated to content 
standards. To submit a web resource users must provide the URL, a title, the targeted audience, and a brief 
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description of the website. Users must also align the web resource by selecting content standards that it addresses. 
This is done the same way that teacher-created and MarcoPolo lesson plans are aligned to content standards. 

Professional Development Calendar 

The Teacher Zone is designed to provide announcements, grant opportunities, and other information that 
teachers can use to increase their knowledge and improve classroom instruction. The professional development 
calendar is included in this section and allows users to create and view professional development opportunities 
across the state of Alabama. To submit an event for inclusion on this calendar the user must include information 
such as the following: the title of the event, a description of the event, the sponsoring organization, the location, the 
date, the cost to attend (if any), and seating availability. JavaScript is used to verify that all required information is 
included before the information is submitted. Once submitted, users are allowed to re-use this information to submit 
events for additional dates; they can modify any of the original information. The user is sent an email message 
asking them to confirm that they submitted an event for inclusion on the calendar. They must click on a link in the 
message to confirm that they submitted the request. To approve events, approvers’ use a similar interface to the one 
the public uses. They have administrative permission, so they can see the status of all submitted events. An event 
can be assigned three different types of statuses: submitted, confirmed, or approved. Once an approver approves an 
event, the event becomes viewable to the public. This is implemented by updating the approved field in the calendar 
table. Designated approvers can edit any event’s information at any time. If the approver decides to reject the 
event, it is deleted from the calendar database table. This section also has a search feature. Users can search by 
category, date, grade, and seating availability. 



Other Features 

The last section, teacher-to-teacher, makes use of freeware. This section offers listservs and forums. 
Mailman, http://www.gnu.org/software/mailman/, is the listserv software that is used. Thirty-one listservs have 
been created for each type of teacher and administrator. For instance, there is a listserv for elementary physical 
education teachers, foreign language teachers, elementary school principals, elementary school counselors, middle 
school math teachers, high school music teachers, etc. The mailman software provides several user interfaces for 
subscribing, viewing and editing subscription information. Users can change passwords, digest messages, and 
unsubscribe, etc. An administrative interface is also available for modifying list configuration information. Forums 
are available for discussion among ALEX users. Ten forums are currently available for discussion. Polls have also 
been setup for users to vote on. These polls serve as a way to get feedback from users on a variety of issues. 

Reports 

Users that have been assigned a user type of administrator have access to reporting tools. In some cases 
additional users are granted access to these reports. There are four types of reporting tools: website usage, 
registered user information, teacher-created lesson plan statistics, and MarcoPolo lesson plan information. 

Website usage tools are available courtesy of Webalizer, http://www.mrunix.net/webalizer/, free software that 
generates charts and graphs from web server log files. Webalizer provides tables and charts, which include statistics 
on the number of hits, files, pages, visits, and Kbytes processed per hour and per day. 

Information on registered users is available for administrators. This tool incorporates a search feature so 
administrators can search for users by names or usernames. Administrators can select a user from the list of search 
results and edit user information. Additionally, administrators can assign administrative privileges. Finally, 
administrators have the option of deleting users from this interface. 

Teacher-created lesson plan information is provided for administrators and designated school system 
contacts. The first interface allows administrators to search for approved lesson plans. A user interface allows 
administrators to search by title, the lesson plan’s author, or the author’s school or school system. A second 
interface is designed for school system contacts. In addition to approved lesson plans, submitted lesson plans are 
included in this report. 

MarcoPolo reports are available to users with administrative privileges. From this screen administrators 
can view alignment information. They can search by username or enter a date range. Search results include lesson 
plans that were aligned by the given user or during the given date range. Accompanying information includes the 
content standard(s) each lesson plan is aligned to along with the date and time of alignment. 
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Database Structure 

The database consists of 22 tables. Included here is a brief description of each table and its function. The 
reference table contains definitions for course and subject codes used in the tblStandards table. The tblStandards 
table holds the content standards for every course of study. The bookmarks table stores links to ALEX and 
MarcoPolo lesson plans; this allows users to maintain bookmarks to these resources on their personal workspace. 
The calendar table is where events for professional development sessions are stored. The group_members table 
keeps track of the users that are group members. The lessonplan table stores the teacher-created lesson plans. It 
includes descriptive information about the lesson plan such as the duration of the plan and the materials needed to 
complete the activity. The steps table contains the procedures associated with a lesson plan. It lists the steps that 
must be taken to complete the activity. The marcopolo table stores information about MarcoPolo lesson plans. The 
messages table stores messages that appear in the personal workspace area. The ranking table allows lesson plans to 
be ranked by users based on usefulness, content, and creativity. It also includes comments. The schools table lists 
all the public schools in the state, so educators can select their school system and school when registering. User 
information is stored in the user table. The variations table holds suggestions for varying or enhancing lesson plans. 
The weblinks table stores URLs that are aligned to content standards. 

Conclusion 

The process used in the creation and maintenance of ALEX has been outlined in this paper. The ALEX 
system is an effort to encourage the K-12 education community in Alabama to improve education by pooling 
resources. The ALEX system continues to grow. Expansion has been built into the ALEX system. Therefore, as 
long as teachers continue to find the education system helpful, they will continue to contribute teacher-created 
lesson plans, variations to teacher-created lesson plans, and web resources. The more resources available on the 
system, the more valuable the system will become to the Alabama K-12 education community. 

Euture work entails adding other types of valuable lesson plans and resources that are similar to MarcoPolo 
lesson plans. An ongoing duty of maintaining the ALEX education system is implementing suggested 
improvements from users to make the education system easier to use. 
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Ensuring Program Success through Learning Communities 

Dan Eastmond 
Mingming Jiang 
Vincent Shrader 
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Abstract 

Scholarship has focused upon the transformational learning fostered by effective online learning 
communities, but has not addressed how faculty of a distance program can use community to sustain persistence to 
degree completion. This includes a variety of learning communities: (1) initial cohorts created during an 
introductory course; (2) communication throughout the program through threaded discussions, synchronous office 
hours, and instant messaging; and (3) course-related communities developed by the online instructors for each 
course. 



Persistence in Distance Education 

The advent of the Internet has seen a plethora of online courses, degree programs, and even distance 
institutions emerge. The Web is available to a majority of households in developed countries and becomes a 
convenient instructional delivery vehicle that meets the lifestyle demands of working adult students. Kember (1995) 
conducted a seminal study of student progress in open and distance universities in four countries that explains 
persistence based on social and academic integration. His model, a significant extension and reframing of earlier 
scholarship, is limited to pre-internet distance delivery modes, examining only course-level attrition. Currently 
completion rates for online courses are usually much better (roughly 80%) than that of independent study (often less 
than 50%) because the student has a greater sense of strong peer and instructor support, structure, pacing, and 
accountability. Even so, the excitement of this technology to access information and provide substantive discussions 
often gives way to the day-to-day grind of near-daily communications, frequent assignments, and the isolation, 
fatigue, and complacency. Scholarship about learning communities has focused primarily on the intellectual 
discourse faculty members promote in these environments - discussion that allows students to formulate and 
articulate their positions, validate or recast those perspectives in light of peer and faculty input, as well as mature in 
their appreciation for the diversity of academic thought. There has been little attention to the impact this learning 
environment has upon persistence - particularly beyond the individual course to the program completion facilitated 
by this technology. 



Learning Communities to Enable Persistence 

This paper explains how a distance degree program or online institution can engender learning 
communities throughout the students experience with the institution in three crucial ways. First, they can create the 
social and academic bonding of students who begin the program together, encouraging the cohort to support each 
other as they move through their studies. Second, the advisor/mentor who guides the student throughout their 
degree program can foster continual peer support as she or he engages students in important discussion not bounded 
by courses. Finally, in each online course faculty members should purposefully create an environment of intellectual 
discourse - engaging the student group in deep processing of course content and knowledge-making. Each of these 
is further elaborated upon below. 

Initial Cohort Seminars 

Creating initial learning community with a cohort group of students entering a degree program is common 
in face-to-face programs. Many distance programs may inaugurate their ostensibly online program with in-person 
meetings, simply to get students comfortable with each other and the faculty members. Several distance institutions 
have created an initial course that fosters the initial formation of a program online learning community. Often 
lasting three to four weeks this introductory course promotes the development of social relationships among peers as 
they explore with the online instructor the challenges of being a distance student, address academic skill 
development, formulate a degree program plan, gain skills in using the library at a distance, and learn how to use the 
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Web platform (learning management system) through which they will be taking their instruction. They are 
encouraged to continue this relationship thereafter. 

Continuous Mentor Involvement 

Most online programs feature a faculty advisor or mentor who works closely with the student throughout 
their program of study, beginning with the introductory course. Successful programs have mentors who regularly 
communicate with students from one semester to another, fostering peer interaction that extends beyond the course 
interaction. They can facilitate these kinds of learning communities by threaded discussions among their cohort of 
advisees, through group online office hours to handle questions and issues that come up outside of regular online 
classes, and through instant messaging where they demonstrate their availability and on-going interest in individual 
student progress. Ludwig-Hardman and Dunlap (2003) give details about how mentors at Western Governors 
University (WGU) play a pivotal role in scaffolding learners’ cognitive and social development through learning 
communities, thereby improving student persistence. 

Content-based Learning Communities and Student Success 

The impact of learning community formation on distance course completion is tremendous. Eastmond 
(1995) proposed the Adult Distance Study through Computer Conferencing model to predict success with online 
study at the course level based on qualitative research of several distance courses. It includes such factors as (1) 
readiness - the personal and environmental elements that prepare the learner for online study; (2) online 
characteristics - factors of the internet environment itself that are similar or dissimilar to conventional education; 
and (3) learning approaches - those strategies that students borrow or invent to successfully negotiate computer- 
mediated instruction. Gunawardena and Duphonrne (2000) found these factors to be valid through a separate 
quantitative research study. The importance of the online learning community to bring about transformational 
learning, is the prominent feature of an online course. Palloff & Pratt (1999) tell that instructors and students need 
to be able to volunteer their opinions, understanding that they will be challenged, but believing in the greater good 
that will result. Through this interplay, all participants will expand their critical thinking, come to appreciate diverse 
perspectives, and construct for themselves a fuller understanding of the topic the group is investigating. Mutual trust 
for social inquiry must be nurtured in all group members by instructor and students alike. 

The Instructor Role in Learning Communities 

The instructor’s plays an essential role in creating the online course learning community. The e- 
moderator’s role needs to shift from presenter of content to “guide on the side’’ - moving away from initial tasks of 
addressing technical and social concerns to facilitating learning: information exchange, issue and practice 
processing, and knowledge construction (Salmon, 2000). The e-moderator’s role best fosters this learning when the 
e-moderator’ s control, authority, and domination of the conference gradually shifts to learners who exert growing 
autonomy over their own learning process. The moderator’s role is to clarify communications, encourage disparate 
opinions, summarize what has been said, keep the group focused on its objective, and weave messages that “find the 
important points, common threads and disagreements to clarify a discussion that has gone off-direction’’ (Winograd, 
2002, p. 56). 



Beyond Course Learning Communities 

Not only are online learning communities important for the intellectual inquiry found in course dialog, but 
that they are also essential for the social and academic integration necessary for distance degree program 
completion. In developing resources for student success in online programs, Duin, Baer, and Starke-Meyerring 
(2001) note that tools for relationship building are the most critical but overlooked part of e-learning. This includes 
such tools as interactive e-portfolios, e-mentoring, and intentional e-learning communities. This latter item, e- 
learning communities, are intentionally set up for learners to interact between themselves throughout their program 
and with instructors - since all of these adult participants have expertise to share. At WGU we have developed areas 
on an electronic bulletin board area for each degree program where students can access key resources and interact 
amongst themselves and the instructor about the competencies being developed and assessed for each domain of our 
competency-based degree program. These learning communities become “a place where learners can test 
assumptions, try out new ideas, and ask difficult questions by way of the company and support of other learners” 
(Bauman, 1997, p.88). By so doing, the institution enables the creation of educationally-rich virtual learning 
communities, similar to the out-of-classroom environment that traditional higher education institutions try to 
engender for enhanced learning, socialization, and affiliation - important means for enabling student progress. 



148 




References 

Bauman, M. “Online Learning Communities.” Trends and Issues in Online Instruction, second annual 
Teaching in the Community Colleges Online conference, Apr. 1999. 
[lehi.kcc.Hawaii.edu/org/tcc_conf97/pres/bauman.html]. 

Duin, A.H., Baer, L.L., & Starke-Meyerring, D. (2001). Partnering in the learning marketspace. 
EDUCAUSE Leadership Strategies No. 4. San Erancisco, CA; Jossey-Bass. 

Eastmond, D.V. (1995). Alone but together: Adult distance study through computer conferencing. 
Cresskill, NJ: Hampton Press. 

Gunawardena, C. N., & Duphorne, P.L. (2000). Predictors of learner satisfaction in an academic computer 
conference. Distance Education. 21(1), 101-117. 

Kember, D. (1995) Open learning courses for adults: A model of student progress. Englwood Cliffs, NJ: 
Educational Technology Press. 

Ludwig-Hardman, S. & Dunlap, J.C. (2003). Learner support services for online students: Scaffolding for 
success. International Review of Research in Open and Distance Learning 4( 1 ). 

[www.irrodl.org/content/v4. 1/dulap.html] . 

Palloff, R.M. & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the 
online classroom. San Erancisco, CA: Jossey-Bass. 

Salmon, G. (2000). E-moderating: The key to teaching and learning online. Sterling, VA: Stylus 
Publishing. 

Winograd, D. (2002). Guidelines for moderating online educational computer conferences. TechTrends 
46(5), 53-57. 



149 




A Miracle or a Menace: 

Teaching and Learning with Laptop Computers in the Classroom 



James Efaw 
Scott Hampton 
Silas Martinez 
Scott Smith 

United States Military Academy 



Abstract 

All freshmen at West Point now have laptop computers to facilitate their learning experience. In this 
presentation we share teaching techniques, lessons learned, and student performance while using laptops in the 
classroom. Presenters will demonstrate techniques to effectively leverage the laptop technology in a way that 
enhances the delivery and efficiency of instruction as well as enhances student learning and interest. Additionally, 
we will present both quantitative and qualitative data comparing laptop classrooms with traditional classrooms for 
the same course. 



Introduction 

Increasingly, college students are showing up to the classroom, not with paper and pencil in hand but with 
laptop computers. Is this new educational tool a miracle or a menace? Is it advantageous or disastrous? How can 
you, the educator, get maximum value out of students using laptops in the classroom and will this technology 
increase student performance? All freshmen at West Point now have personal laptop computers to use in class. 
Several instructors for the General Psychology course for all freshmen are currently incorporating classroom 
learning activities and strategies to leverage the technology tools available with the laptop computers. The purpose 
of this study was to examine teaching techniques, lessons learned and student performance during the integration of 
laptop computers to teach and learn psychology in the traditional face-to-face classroom to enhance teaching 
practices and efficiency and positively influence student learning and attitudes toward psychology. 

As we present our findings, we will try to highlight the influence of laptop technology on student 
performance (learning) and attitudes (motivation/interest) as well as the impact on teaching efficiency. Even 
miraculous results could be easily discounted if using such technology is a menace for teachers to integrate into their 
daily practices. We developed a short acronym to categorize the impact that laptop technology had on student 
motivation/interest (M), teacher efficiency (E) and student learning (L). Our intent, then, was to integrate laptop 
technology (T) in such a way to prevent the teacher from literally having a technology ‘MELT-down’ from another 
menacing ‘great idea’ about integrating technology. 

We initially attempted to integrate the laptop technology into the classroom during our summer teacher 
training sessions. We soon discovered that the returning faculty members participating in these training workshops 
were not very motivated to integrate the technology. Many instructors felt uncomfortable with teaching while 
students were actively pounding the keyboards to take notes. Additionally, many faculty members believed that 
many students would quickly engage in activities not related to the class. Giving up this ‘control’ in the classroom 
was not acceptable to selected faculty. One faculty outwardly stated “My students won’t be using those laptops in 
my classroom.” Eaculty’s initial attitudes had an adverse impact on our attempts to get students using laptops in the 
classroom. 

Another critical limitation influencing faculty utilization of laptops was whether faculty members actually 
owned a laptop. During our faculty training, we used ‘loaner laptops’ and then turned them in before classes started. 
We soon discovered that faculty members were much more likely to attempt utilizing laptops in the classroom if 
they actually had their own office laptop. Eortunately, we were able to acquire 4 laptops for the 4 instructors who 
participated in this study. Consequently, when we issued laptops to the six new instructors on our faculty this year, 
which is the semester after this study was conducted, all 6 instructors aggressively worked to integrate laptop 
technology into their classroom strategies. 

Another hurdle to laptop integration involved our learners. Because this was the first year that students at 
the Academy had laptops in the classroom, the students were also quite speculative about the utility of this 
newfound technology in the classroom. They definitely saw advantages to doing email and Instant Messaging in 
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class, but using it as a tool for learning was not evident at first. Several students simply resorted to paper-pencil 
note-taking with a closed laptop beside them. 

Method 

Participants. 

The study involved 10 General Psychology instructors who taught 527 college freshman students in 30 
classroom sections (average of 17 students per section) at West Point. Students were randomly assigned to their 
classroom sections. The instructors and students used the same textbook, curriculum, learning objectives and 
exams. 

Design. 

A quasi-experimental design examined the effects of utilizing the laptop technology on classroom 
practices, student learning and attitudes. Instructors (six) in the control group did not incorporate laptops into their 
teaching strategies for 22 student sections, while four instructors in the experimental group used the laptops to 
facilitate their learning activities in class for their 8 student sections. The four instructors volunteered to participate 
in the study. These instructors varied in experience. One of the instructors has a PhD in education and 5 years of 
teaching experience. Two of the other three remaining instructors were first-year teachers while the final instructor 
had three years of teaching experience. All three of the teachers had masters degrees in psychology. 

Measures. 

Six exams (combination of multiple-choice and short answer) and a final exam (100 multiple-choice with 
reliability of r = .81) assessed student learning of course material. All exams were the same for all students (taken at 
the same time) and instructors held calibration sessions before the exams to maintain consistency in grading criteria. 
A typical end-of-course survey assessed student attitudes toward the course and the integration of technology. 

Because the Academy was in the process of configuring classrooms for wireless technology, we used three 
different types of technology classrooms to best simulate a wireless technology environment. One instructor had 
access to a wireless ready classroom, one instructor used a standard desktop lab that simulated wireless capability 
with a server, and two instructors used the laptops in the traditional classroom with limited wireless capability. 
Interestingly, even in the most wired ‘hi-tech’ classroom, several students refused to use their laptops and chose to 
stay with the traditional paper-pencil note-taking. 

‘Bringing learning to life’ classroom techniques 

What follows are various techniques and strategies that were used in the laptop sections to ‘bring learning 
to life’ to enhance student interest and motivation and various techniques to facilitate practice and feedback. By 
having students use the laptops in the classroom, the teacher has a much wider range of instructional and classroom 
tools available. The advent of wireless networks across campuses fits hand-in-hand with laptop use. A wireless 
network is not just for the students to use in the library or their dormitories, but it should also be used by the students 
and the instructors in the classroom. The instructor now has the flexibility of simultaneously having all their 
students go to the World Wide Web during class and going to specific web sites that are related to class. The web is 
full of interactive and interesting sites that quickly and efficiently (E) demonstrate key classroom concepts. The 
web sites can be used as a topic introducer, a discussion generator and point solidifier. For example, the instructor 
can have the students go to a particular web-site that will illustrate or introduce concepts and phenomenon that are 
related to concepts to be covered later on in class. These same web sites often spark interesting discussion and 
debate which really engages the students’ interest (M). Similarly, after just covering an abstract psychological 
concept, the class can go to a web-based exercise that illustrates the concept in action. Finally, often topics come 
up for which the instructor did not prepare or are outside the scope of class but that can quickly be looked up on the 
web. Quick, in-class research is a great tool. For example, suppose the days’ class is on sleep deprivation. The 
instructor could quickly say, “Okay, student X, I want you to find who holds the record for the most consecutive 
hours without sleep and student Y find me five symptoms of sleep deprivation.” Within five minutes, the instructor 
and the class would have the answers. This easy, quick activity (E) increases the students’ motivation (M), keeps 
their interest by giving them in-class activities, and ultimately contributes to learning (F). 

If there is one activity in which today’s generation enjoys engaging, it is computer games. Since this 
motivation (M) already exists, the key is to harness this interest and bring it into the classroom. For example, when 
teaching the experimental method, one could use any number of computer games to “test” hand-eye coordination of 
your students. Of course, it would probably be necessary for you to bring in the “pill” (perhaps a green M&M®) that 
you have “developed” to improve hand-eye coordination. The instructor could have the students design the study. 
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develop the hypothesis, figure out how to choose participants, divide the participant into the control group and 
experimental group, etc. Then have the students do the experiment, have them take the pill and play the game on 
their computer. Within an hour-long class, the students can go through and execute the entire experimental method 
to include, perform the experiment (on the laptop), gather the data, analyze the results using programs on the laptop 
computer such as Microsoft® Excel and even talk about methodological problems associated with the classroom 
“experiment.” 

Another great use for the computer is digital media. Nothing captures the students’ attention like a clip 
from a popular movie, television show or funny commercial. One may ask, “Well, why is there advantage with 
laptops and digital media? I use digital media from my room computer already.” The big advantage of digital 
media is in the area of motivation (M) and learning (L). With digital media, the instructor can share with the 
students via email or other sharing techniques, the digital clip that was used in class. We have found that these 
students then go back to their rooms with their laptops, or take their laptops home, or to the coffee shops with their 
friends to study and they share the video clip with others. We have found that they are not only sharing the clips but 
they are then explaining to their friends, parents, boy/girlfriends, etc. the principle or concept tied to that video clip. 
The digital media brings the learning (L) and motivation (M) outside the classroom, and any time that we can get 
our students excited and talking about our discipline outside the classroom, it is “a good thing.” 

Lastly, nearly all introductory psychology textbooks now come with a CD-Rom enclosed. Most instructors 
and students seemingly ignore this valuable learning (L) tool. Perhaps the most valuable tools available for the 
classroom on these CDs are the simulations, demonstrations, and exercises which can often be used in the same 
manner as the web-based exercises discussed previously. These exercises are usually fun and interesting and 
comments from the students show that they motivate (M) their learning (L). Before laptops in the classroom, 
instructors could perhaps assign these exercises as homework, suggest that the students look at a particular 
demonstration, or best case, demonstrate it themselves on the classroom computer for all to watch. The instructor no 
longer has to wonder whether the student did the assigned simulation nor does the student simply have to watch as 
the instructor performs the exercise in front of the class. Now each student can put their own CD into their own 
laptop and do the class exercise while the teacher circulates the room and monitors everyone’s progress. These 
exercises, demonstrations and simulations typically only take up 5-15 minutes of class time and at the completion 
each student has had a common experience which can lead to rich classroom discussion. Additionally, by putting 
into practice concepts just studied, the CDs tend to cement the concepts into the students’ mind. They now have a 
visual, auditory and tactile tie to what they have read for homework the previous night and that is going to be 
covered in class that day. 



Practice and feedback classroom techniques 

In this section we include various techniques for effectively utilizing laptops in the classroom. These 
techniques really affect the efficiency (E) of classroom procedures while promoting learning through responsive 
feedback enabled by technology. 



Digital Syllabus 

Provide an editable, digital syllabus. Provide a detailed syllabus (skeleton outline of your lesson plan) 
which lays out the “big-picture” lesson objectives as well as the lower-level performance objectives which spell out 
the concepts with which the student must be familiar before attending class. By providing the digital syllabus, 
students can do their lesson preparation on their laptop computers by adding their notes in a different color right into 
the course guide that we provide. Taking notes in a different color makes it easy for the student to follow-along in 
class, and also makes it easy for the instructor to walk around and see which students have prepared for class. While 
in class, students can use a third color to update or correct their notes as required and to highlight those concepts 
upon which the instructor focuses. By the time class is complete, the student has produced a legible resource for 
future use and study. 



“Beam Solutions” 

Use wireless capability to “beam” student homework solutions on the board for review. The wireless 
network allows for quick transfer of homework or in-class assignments to a digital drop box which can be accessed 
by the computer to which the overhead projector is attached. Most classrooms have either a chalkboard or dry-erase 
board behind the screen. Once in the digital drop box, the instructor can “beam” any student’s solution onto the 
board where it can be annotated to illustrate either common pitfalls or exemplary work. Since the source file was 
created on the laptop, then the owner of the on-screen assignment can correct his work as the teacher does. At the 
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end of the exercise each student has a correct example of a given type of problem. We have found that for 
classrooms that are not wireless, the student can save her work on a disk or CD and then the teacher can simply load 
it up on the classroom computer. Compare this technique to doing in-class work on chalkboards. Not only can the 
student and instructor cover multiple examples (E), when simply solving problems or doing homework on the 
chalkboard, any feedback or corrections end up as a pile of dust in the chalk tray when erased at the end of class. 
Finally, using actual student responses seems to be more meaningful (M) to our students then “canned” examples 
that may or may not cover specific difficulties that our students may be having. 

Software interaction 

Incorporate use of discipline-specific software in classroom exercises. Our instruction on correlational 
research incorporates an in-class exercise in which we try to correlate video game exposure with academic 
performance. Rather than cite some study which the average student cannot relate, we used a spreadsheet program 
to calculate the correlation coefficient and its associated p value for the students in our class. Using personalized 
data really brings what can be “dry” material to life for students (M) because it is their data. At the same time it 
allows instructors to cover higher-level concepts. Finally, the student gets practice using a tool to calculate an 
important statistic which she will then interpret. At the end of class, the student can “take away” the data set and 
have something for further study or reference as desired. 

In-class group projects: 

Use laptops to create in-class group projects or presentations. Many phenomena are explained by different, 
competing theories. Since each student has laptop, instructors can assign each group to prepare a five-minute 
presentation covering a theoretical perspective. Students can use the presentation software on their laptop to create 
their brief that will be given to the rest of the class. After sufficient preparation time, each group can “deposit” their 
presentation in the digital drop box (or save it to a disk) so that it can be displayed on screen as they give their 
presentation. This technique allows students to practice speaking skills, get detailed study of the theoretical 
perspective they were assigned, get a presentation on the other theoretical approaches presented by the other groups, 
and take away electronic copies of all the presentations created by their classmates. 

Results 

We assessed students’ performance using in-class exams. Students of instructors who integrated laptop 
computers into their classroom strategies (experimental group) scored significantly higher (p < .05) on all 6 exams 
and the final exam than students of instructors who used traditional instructional and note-taking methods (control 
group). 

We assessed students’ attitudes through End-of-course surveys. We categorized the 42 questions into one 
of the three dimensions examined in this study, which included student motivation/interest (M), instructor efficiency 
(E) and student learning (L). Nine of the 15 questions (60%) related to student motivation and interest were rated 
significantly more favorable (p < .05) in the technology sections. For example, on a Likert scale of 1 (strongly 
disagree) to 5 (strongly agree), students in the technology sections rated instructors significantly higher on the 
following question categorized for student motivation (M): “My instructor was enthusiastic and energetic when 
presenting course material.” A similar trend in favor of the technology sections was also evident in 3 of the 5 
questions related to teaching efficiency (E) and 9 of the 22 questions (39%) related to student learning. Overall, 
students in the technology sections (experimental group) rated questions on this survey more favorably than the 
traditional sections (control group) and 21 of the 42 questions (50%) were statistically significant (p < .05). 

We also included two open-ended questions on our end of the course survey. The two questions were: 
“What did you like about using your laptop in class?” and “What did you dislike about using your laptop in class?” 
We took all responses to these questions and categorized the common responses. 

For “What did you like about using your laptop in class?” the comments fell into two main categories. Of 
the students that used the laptops in class, 73% made comments that fell into the category of ease in note-taking or 
ease of organizing/reading their notes. These comments highlight the efficiency aspect of taking notes in class. 

Some sample comment include: 

• “Typing is a faster more efficient way of note taking.” 

• “I could take notes faster & keep my stuff organized.” 

• “I had all my notes (on laptop) & could add to them when I felt appropriate, & they were kept very 

neat, organized & readable on the computer. In a notebook, they possibly would have been lost or 

neglected, or less legible & organized.” 

• “I did not have to shuffle through papers. All I had to do was hit page up or page down and I had 

the information I needed. It was also neater than my handwriting so i could read it easier and faster.” 
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It is faster, neater, and more efficient in taking notes.” 



24% mentioned other uses of the laptop such as computer and internet exercises to include web sites, CD 
ROM exercises, games. These comments highlight the motivation aspect of using laptop for classroom activities in 
the classroom. Some sample comments include: 

• “I like some of the experiments that we did on the CD in class.” 

• “We were able to access various websites & tests that related to the material.” 

• “I liked the group time when we used the computer to type things down & then present them in 

class. . .it was a VERY effective learning tool.” 

• “It allowed us to use simulations to do different experiments during class.” 

• “I liked using my laptop because we could use the internet to look at stimulations relating to the 
course material.” 

For “What did you dislike about using your laptop in class?” the comments also fell into two main 
categories. 50% of those students who did use the laptop did not like carrying it back and forth to class — mainly 
because they felt that it was heavy and cumbersome to bring to class. Sample comments include: 

• “Pain to carry to class.” 

• “Bulky to carry around.” 

• “Lugging it around from class to class.” 

• “Carrying the laptop is always a chore.” 

• “I dislike the fact that we had to haul the laptops to class. I hate carrying my laptop around.” 

32% of those used the laptop made comments that fell into the category of learning style and typing skills. 
These students still preferred to take notes with pen and paper because they a) did not know how to type, b) did not 
like the fact that they could star, draw arrows to, or make notes in the margins, or c) they felt the learned better by 
writing notes by hand. Sample comments include: 

• “I feel as if I learn better by writing things down as opposed to typing them.’Tt is hard to enter 
charts and graphs into the laptop with efficiency.” 

• “I found it much easier to add notes to the margins and maneuver through the course guide in 
paper format.” 

• “Sometimes I could not type fast enough to take notes in class.” 

• “It takes me longer to type than write and sometimes its easier to draw diagrams than write 
something out in words.” 



Lesson learned 

This section will cover some of the practical aspects of using laptops in class. Specifically it covers 
classroom management techniques, some dangers and pitfalls, and finally some of the benefits of using laptops in 
class. 

Classroom Management Techniques: (E) 

Batteries only - no cords. With a classroom full of computers, battery power is a must. If each student had 
to plug in their laptop, some form of accident would be inevitable. The instructor will eventually trip over a cord 
and hurt himself or the students $1500+ investment. 

Provide an editable, electronic syllabus. This will allow students to do their before-class preparation and 
in-class note-taking as described above. 

Screens down during movies. Require students to put their screens down during movies so that they don’t 
interfere with another student’s ability to focus on the screen, and so they “get” what you intended during the film. 

Employ a “No Outside Work” Policy in class. Our students cannot skip class without punishment, so there 
has always been an occasional tendency to try to complete requirements for another course during class time. 
Enabled by wireless access to the internet and network folders, the laptop merely provides a new means to try to do 
this. Don’t be afraid to tactfully address violators. A direct but tactful request is generally all it takes to get students 
back on track. 

Publish a set of “Classroom Rules. ” We find that most students want to do the right thing most of the 
time. Providing a list of norms will ensure that your students know what is and what is not allowed in class. 
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Dangers and Pitfalls 

Surfing the Web. Initially, this is the biggest challenge to using laptops in class. Fortunately, it is easy to 
spot because students are decisively engaged with the computer, but there is not much keyboard activity involved. 

As mentioned above, a direct but tactful request is generally all it takes to get students back on track. 

Instant Messaging. This is also a big challenge during the first couple of classes. Unfortunately, it is more 
difficult to detect this activity because it does resemble taking notes. By moving around the class and behind the 
students, you will keep most folks on track. Another technique is to look at which programs or applications are 
active and which ones have been minimized. Again, a direct but tactful request is generally all it takes to get 
students back on track. 

Legitimacy of websites. Using the internet is one way to spark interest in your topic, but students must not 
leave class thinking that every site visited in class is “scholarly.” For example, a web-site offers little applets to help 
students determine the “sex” of their brain. While useful for generating classroom discussion, that web applet 
should not be confused for a scholarly resource. Other sites have similar issues. 

Trying to have a computer exercise for everything. The laptop is a tool that can be used by the instructor to 
facilitate learning, but it should never be the focus of the class. Some topics are better covered in small-group 
discussions, some concepts are better brought to life with other teaching techniques such as skits. Sometimes the 
laptop is not the best teaching and learning tool. The key is to use the most appropriate medium for the material to 
be covered (E). 

Benefits of Laptops in the classroom: 

Using computers in the class promotes computer literacy for life-long learning (L). The common 
assumption is that today’s college freshmen are computer literate, but we have found that they are very good at 
using the internet, instant messaging, and playing games. Many of them are not very good at using the software 
applications most useful for schoolwork. Using computers in class “encourages” them to learn how to effectively 
use the tools that they will be using the rest of their college “careers” and beyond. 

Using laptops in the classroom also increases efficiency in practice and feedback in the classroom (E). 

Since students can easily submit their assignments and have them shown on screen, instructors can cover more 
assignments in class than would be feasible with transparencies or on the chalk board. 

Einally, using the laptops so each student can play a game, or go to a website, or do an on-line experiment 
does really seem to spark student interest (M) in psychology than demonstrating those same items on the big screen 
or other traditional techniques. In the end, getting students excited about learning is what we are all trying to 
accomplish. 



Discussion 

Laptop computers are coming to the classroom whether you are ready or not. Will you be ready to turn this 
new yet prevalent technology into a benefit for you and the learner or will it simply be a distraction? In this paper 
we have shared with you several useful techniques for both bringing learning to life and for feedback in the 
classroom that we have discovered after one semester of teaching introductory psychology to a classroom full of 
students with laptop computers. We have found that these techniques 1) increase the students’ motivation (M) about 
the topics covered, 2) add efficiency (E) to the classroom in both note-taking and amount of material covered and 3) 
seem to actually increase the students learning (L) (as was supported by the increased tests scores of those students 
in the laptop vs. no laptop classrooms). 

Employing the laptop computer in the classroom is not all easy. It is work. Often this means revising 
nearly all your lesson plans and thinking about your topic in fresh, innovative ways. As discussed in the paper, there 
are many dangers and pitfalls that can only be avoided by utilizing different classroom management techniques. 
However it has been our experience that in the long run the benefits outweigh the costs and we wish you luck in 
your endeavor to effectively utilize laptops in the classroom. 
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A Case of Supporting Distributed Knowledge and Work in the U.S. Navy: 
The Challenges of Knowledge Management to HPT 

Michael A. Evans 
Indiana University 

Abstract 

Knowledge to maintain and troubleshoot complex systems is a vital asset to organizations. Consequently, 
the Navy has chosen to adopt precepts and practices from knowledge management (KM). From a human 
performance technologist’s perspective, this raises two questions: 1 ) How to think about knowledge, particularly 
within a distributed setting? 2) How to develop solutions that allow for distribution across organizational 
boundaries? Drawing from compatible perspectives — communities of practice, activity theory, and institutional 
theory — I recommend thinking of knowledge as a process of participation in practice. Consequently, interventions 
need more creatively account for social and cultural features of performance. To illustrate, I present a case of 
distributed knowledge and work between virtual teams in the U.S. Navy. Specific knowledge management strategies 
and solutions are offered in conclusion. 



Introduction 

Over the past year I have been conducting user, task, and context analyses as part of a larger project to 
develop a knowledge management system for the U.S. Navy. In this piece I share this experience as well as point out 
a few of the challenges encountered. Ideally, this account will provide the opportunity for others to compare their 
work, contributing to the discussion of how best to incorporate knowledge management tools and tactics into the 
practice. 



A Knowledge- Centric Approach to Performance Improvement 

The U.S. Department of Navy (DON) is in a technologically-intensive transition period. In essence, to 
counter a continued downsizing of on-board personnel the agency intends to leverage what is perceived as the added 
value of tacit, expert knowledge. Taking a knowledge management position, the goal is basically to improve 
efficiency and effectiveness. An explicit indication of this desire for change in knowledge, work, and technology is 
captured in Goal 4 of the recently released Information Management & Information Technology Strategic Plan 
FY2002-2003 (Department of the Navy Chief Information Officer, 2003). The intent is to 

[i]mplement knowledge management. . .strategies to facilitate collaboration and 
information sharing that optimize strategic and tactical decisions, resulting in more effective and 
efficient mission performance. Knowledge management... ojfer[s] the potential to significantly 
leverage the value of IT investments and the intellectual capital of our people. Information 
technology and information management are essential, but alone are insufficient to achieve 
information superiority. (p.20) 

Consequently, one approach to realize this strategy is to develop a performance support and knowledge 
management system. The system would aid at a distance the collaborative troubleshooting actions of military and 
civilian technicians and engineers maintaining a complex system aboard U.S. Navy warships. To undertake this task, 
designers and researchers from the Knowledge Acquisition and Projection Lab at Indiana University are 
participating in the Knowledge Projection Project - a joint undertaking with Naval Sea Systems Command 
(NAVSEA), Naval Surface Weapons Center (NSWC) Crane Division, EG&G Technical Services (a government 
contractor) and Purdue University. The anticipated system is intended to leverage both tacit knowledge and 
advanced information technologies. Accordingly, the design team at lU expects to exploit KM precepts and 
practices to positively impact key organizational variables, viz., total cost of ownership, maintenance and 
troubleshooting efficiency and effectiveness, and fleet readiness. 

The system being proposed is meant to support knowledge and work distributed across individuals and 
organizational boundaries. Intriguingly, this situation presents two challenges on a conceptual level. These 
challenges are: 1). how to think about knowledge and work in a distributed fashion and, 2) how to account for the 
social and cultural features of performance. As described below, social features become relevant as performance is 
essentially a collaborative and distributed practice across specialized work units; likewise, cultural features become 
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relevant as coordination cuts across occupational identities and boundaries, defined, for example, by status in the 
organization (military vs. civilian) and functional role in the end-to-end process (primary maintainer vs. first-line 
support). My challenge has been to try to address these issues from the perspective of a performance technologist. 

Taking Account of Distributed Knowledge and Work 

Interestingly, knowledge, work, collaboration, and culture come into play in this scenario of distributed 
knowledge and work. This effect has motivated an exploration of alternative frameworks for analyzing performance. 
My inclination is that conventional views from behavioral psychology and cognitive information processing 
conservatively think about knowledge as an object, something to be codified and transmitted independent of context. 
Consequently, when HP technologists prescribe interventions with this perspective in mind, they tend to focus on 
ways to support for this transfer. Moreover, the recipient of treatment is often the individual in isolation. Examples 
include textbook cases, job aids, intelligent tutoring systems and web-based training. The shortcomings of these 
traditional views and approaches are that they do not easily permit for 1) the conceptualization of knowledge as a 
process in social context and 2) the development of interventions at levels of organization that inevitably encounters 
cultural and institutional elements. 



Level of Intervention 





Individual 


Group 


Organization 




Cell 1 


Cell 2 


Cells 




Computer-Based Training 


Electronic Performance Support 


Business Process Re- 




Self-Directed Learning 


Systems 


engineering 




Classroom Instruction 


Groupware 


Process Improvement 


Object 


Job Aids 


Learning Objects 


Knowledge Management 


Cases 


Distance Support 
E-Learning 


Systems (Technical-Object 
View) 

Competency Modeling 
Organizational Storytelling 




Cell 4 


Cells 


Cell 6 




Apprenticeship 


Action Learning 


Socio-Technical Systems 




Mentoring 


Communities of Practice 


Knowledge Management 


Process 


Coaching 


Computer Supported 


Systems (Social-Process View) 




On-the-Job Training 


Collaborative Work (CSCW) 
Role-plays & Simulations 


Balanced Scorecard 
Constellations of Practice 



Table 1. A scheme for proposing alternative frameworks for analysis. 

Within each cell are interventions drawn from a specific theoretical orientation, denoted by the view of 
knowledge and intended level of intervention. For example, computer-supported cooperative work draws from 
activity theory while knowledge management system (social-process view) draws from institutional theory. Adapted 
from original work by Kalman and Schwen (2000). 

To counter, I have juxtaposed three perspectives — communities of practice, activity theory, and 
institutional theory — that more readily permit for the kind of analysis and design anticipated. Getting bogged down 
in theoretical abstractions is not the purpose. On the contrary, my position is that these theories lead to intervention 
options not available in the conventional HPT toolkit. To this end. Table 1 provides the backdrop for an exploration 
of alternative views and subsequent interventions. Basically, my argument runs likes this: “If increasingly we 
encounter organizational problems where groups are working together collaboratively on a knowledge-intensive 
process, then it may be beneficial to explore and incorporate alternative perspectives that permit attendance to 
previously overlooked features of organization.” 

Communities of Practice 

From a communities of practice (CoP) perspective knowledge and work occur inseparably from the social, 
situated contexts of organization. A relevant conclusion drawn is that if knowledge and work are conceptualized as 
an act of belonging to and participating in a community, then identities (occupational, not personal) should be key 
vehicles by which one comes to know a practice. Interestingly, Wenger (2000) maintains that identity is crucial to a 
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social learning system for three reasons. First, identities combine community-defined competence and individual 
experience into ways of knowing. Second, one’s ability to deal productively with community boundaries depends on 
an ability to engage and suspend identities. Finally, identities are the living vessels in which communities and 
boundaries become realized as an experience of the productive world. 

Noticeably, identity as a concept is crucial in this project as sailors and civilians interact on demand to 
accomplish complex, orchestrated tasks. Although members of each community can be labeled ‘technician,’ the 
knowledge they offer and which is made available by the community differentiates, rather than integrates identities. 
Identity, then, can be seen as a social feature of organization that impacts performance. 

Activity Theory 

According to the principles of activity theory (Engestrom; 2000), an activity is a coherent, stable, relatively 
long-term endeavor directed toward an identifiable object, or goal. Within the fundamental activity system, a subject 
(individuals, groups, or other activity systems) use tools to achieve this goal. For instance, sailors use technical 
manuals, electronics terminology, and measurement instruments to complete the goal of troubleshooting a piece of 
gear or equipment. Moreover, an activity can be understood only within its culturally situated context, defined by 
the community, the rules they enact, and the division of labor they enforce. Interestingly, one area where activity 
theory differs noticeably from communities of practice is the emphasis of tensions, or contradictions, that arise 
within and among activity systems. 

Within the Navy case, a principal concern is with contradictions that arise between sailors and civilians 
along lines of identity, knowledge, and community membership. One clear tension exists between obligations sailors 
have toward the Navy and toward the ‘technician’ community. A challenge here for HPT is to determine how best to 
handle these tensions because, from an activity theory perspective, contradictions can at times spur learning and 
innovation. 

Institutional Theory 

Institutional theory (IT) is the third perspective to add value to this analysis. Two constructs found in IT 
especially attractive are power and meaning. That is, institutions, when ‘taken-for-granted’ exert power on social 
actors by legitimating structures, rules, and procedures. In contrast, meaning is the method by which actors change 
institutions by specifying innovations then Justifying them by appealing to legitimacy (Scott, 1995). 

From analysis, one instance of conflict found between power and meaning involves the training and 
development of sailors. Whereas the Navy as an institution exerts power to establish the right blend of sailor and 
technician in a recruit, sailors themselves instill meaning into the role of sailor-technician by creating mechanisms 
(such as ‘refresher courses’) to maintain what is perceived as the required balance. 

In summary, communities of practice, activity theory, and institutional theory together bring fresh insights 
to the problems faced by HP technologists when contemplating KM ideas and initiatives. What follows is a case of 
distributed knowledge and work within this context. 

Troubleshooting a Fault: A Case of Distributed Knowledge and Work in the U.S. Navy 

In the U.S. Navy Electronics Warfare (EW) technicians are tasked with the operation and maintenance of a 
ship defense system referred to as the “Slick-32.” Put simply, the “Slick-32” acts as a sophisticated radar detector 
for over 200 ships in the fleet, detecting and identifying any object that enter a ship’s operational horizon. 

Whereas many prescribed troubleshooting procedures for the “Slick-32” are documented thoroughly there 
is a certain class of problems that fall outside this sphere. Consequently, these problems, or “faults,” often require 
EW technicians request assistance from a team of waterfront civilian technicians and engineers available to the fleet. 
One possible limitation on a sailor’s knowledge, which would necessitate this request, may be that she is familiar 
only with the particular system on her ship. Likewise, she may have spent most of her sea duty following precisely 
detailed procedures that permit little improvisation. What differentiates waterfront, civilian SMEs from sailors are 1) 
although many have performed military service, they are now classified as “civilians”; 2) they have on average had 
several more years experience working on all variants of the “Slick-32”; and 3) they have troubleshot any number of 
“undocumented” or “unidentified” faults over those years. These technicians and engineers, on the other hand, do 
not possess the detailed knowledge of any one specific system and its unique operating environment prior to a 
technical assistance request. To contrast the two, when a sailor is troubleshooting her system she is restricted in 
knowledge by training, the particular ship, and the specific piece of gear. On the contrary, the waterfront technician 
is able to draw from a rich store of knowledge accumulated from experience. Another way to look at this is to say 
that the sailors rely heavily upon explicit, organizationally-mandated knowledge. This knowledge is formalized in 
the shape of curricula and technical manuals that standardize practice across the enterprise. Technicians, on the other 



158 




hand, draw from a deep store of tacit, highly-individualized knowledge to conduct their troubleshooting work. This 
knowledge has been gained by working hand-in-hand with hundreds of shipboard work units and their systems. 

Thus, in this intriguing configuration, collaborative troubleshooting across structural and cultural boundaries is 
further complicated by a disparity in knowledge types being drawn upon and exchanged. 

Potential Interventions from a Knowledge Management Perspective 

While a conceptual reframing has been a much needed first step, development of appropriate interventions 
is the ultimate requirement for this project. Consequently, below is offered a sample of tentative knowledge 
management strategies and solutions we have come up with. Each item in the list presents a generic KM category 
first proposed by Earl (2001), followed by project-specific solutions. Table 2 provides a quick reference to see how 
strategies and interventions have been targeted to each of the three user groups. 

What should be kept in mind, though, is that these solutions cannot be pondered independent of the analysis of 
knowledge and work. That is, occupational identities, contradictions between work units, and institutional power 
and actor meaning should inform any intention to intervene. These concepts are the context for the distributed 
knowledge and work being supported by the following solutions: 

Knowledge Bases (“Systemized, Codified Expertise” ): capture SME knowledge and make it available to sailors. 

Codification of engineering know-how, technical expertise, and overall best practices can be accomplished via a 
Case-Based Reasoning (CBR) system, a High-Performance Knowledge Base (HPKB), and a Best Practice 
knowledge base. 

Knowledge Directories (“Who, What, Where’’): record and disclose who in the organization knows what; SMEs 
throughout the organization are made accessible for advice, consultation, or exchange via Subject-Matter Expert 
Directories & Profiles and Skills by Experience Matrices offered over an intranet. 

Knowledge Flows (“Process Improvement"): unrestricted provision of decision-relevant and contextualized 
knowledge at point-of-need to increase efficiency, learning, and adaptation; knowledge can be reused at 
shipboard level via Shared Databases of Logs, After Action Reviews, and Product & Logistical Information, 
and leveraged at the waterfront level via Shared Databases of ‘Customer’ Satisfaction Information. 

Knowledge Assets (“Rent-Producing”): overtly and explicitly concerned with both protecting and exploiting the 
Navy’s knowledge or intellectual assets to produce revenue streams; “intellectual property” is exploited via 
Training Curricula, Systemic Expertise and various forms of “Intellectual Property”. 

Knowledge Pooling (“Networked Communities”): use of technical and social structure to exchange and share 

knowledge interactively as an interdependent network; communities can be designed and maintained to address 
a specific business purpose via Collaborative Problem-Solving & Support Systems, Knowledge Moderators, 
Virtual Teams and Discussion Eorums. 

Knowledge Exchange (“Real and Virtual Spaces”): interaction and discussion promote discovery and exchange of 
(mostly tacit) knowledge; contact and activity can be facilitated via Virtual Shore-based Tours or Visits, Virtual 
Break Rooms or Ship Visits, and Virtual Classrooms, Visits or Conferences. 

Conclusion 

In the end, the goal here has been to offer just one possible path that HP technologists might take to meet 
the theoretical and practical challenges presented by knowledge management precepts and practices. Thinking about 
knowledge as a process and contemplating interventions at collective and organizational levels are not easy tasks, 
especially given our conventional roots. Eortunately, there are emerging perspectives that may greatly facilitate the 
HP technologists venture into this arena. My suggestion is that we begin to use these, and related theories to inform 
and improve development. 
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Knowledge Bases 
( “Systemized, 
Codified Expertise” } 


Knowledge 
Directories (“Who, 
What, Where") 


Knowledge Flows 
( “Process 
Improvement” ) 


Knowledge Assets 
( “Rent-Producing ”) 


Knowledge Pooling 
( “Networked 
Communities ”) 


Knowledge Exchange 
(“Real and Virtual 
Spaces”) 


Sailors 


Text IR; CBR; 
HPKB; Concept 
Maps; 


Subject-Matter 
Expert Directories & 
Profiles; People 
Einders; Technical 
Digests; News; 
Portals; Information 
Channels 


Shared Databases of 
Logs and After 
Action Reviews; 
‘Electronic 

Maintenance Activity 
Support’ 


Training Curricula; 
Task Eorce Excel 
Maintenance 
Expertise 


Collaborative 
Problem-Solving & 
Support Systems; 
Knowledge 
Moderators; Virtual 
Teams; Discussion 
Eorums; Sociable 
Media 


“Virtual Shore-based 
Tours or Visits” 


SMEs 


Text IR; Best 
Practice KBs; CBR; 
HPKB 


Subject-Matter 
Expert Directories & 
Profiles; People 
Einders; News; 
Portals; Information 

/-n 1 


Shared Databases of 
Technical Product & 
Logistical 
Information; 
‘Electronic Depot 

n . 1 


Systemic Expertise; 
“Refresher Courses” 


Collaborative 
Problem-Solving & 
Support Systems; 
Knowledge 
Moderators; Virtual 

'T* 1-^ • 


“Virtual Break 
Rooms or Ship 
Visits” 


Engineers 


Best Practice KBs 


Skills X Experience 
Matrices; News; 
Portals; Information 
Channels 


Shared Databases of 
‘Customer’ 
Satisfaction 
Information; 
‘Electronic 
Intelligence, 
Eeedback & Analysis 
on Equipment & 
Support Materials’ 


Commercialization 
of “Intellectual 
Property”: 

Production Expertise, 
Test Sets, Technical 
Specifications & 
Manuals; Patents 


Knowledge 
Moderators, Virtual 
Teams; Discussion 
Eorums; Sociable 
Media 


“Virtual Classrooms, 
Visits or 
Conferences” 



◄ 

Technological Social 



Socio-technical Emphasis of Strategies & Interventions 

Table 2. A knowledge management design matrix for the Knowledge Projection Project. Notice that strategies and interventions are aligned along a spectrum 
emphasizing either more technical or more social aspects of knowledge management. 
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The Map is not the Territory: 

A Constructivist Application of General Semantics to the Design of 

Educational Texts 

James E. Gall 

University of Northern Colorado 

Abstract 

Constructivism has had an impact on the field of educational technology. However, an objectivist 
perspective still dominates much of our practice. To make constructivism a viable approach, we need to develop 
tools and methodologies to put it into practice. The field of General Semantics has a long history of examining texts 
from a non-objectivist perspective. Ai such, it provides us with valuable insight into how we can create educational 
texts from a constructivist viewpoint. 



Introduction 

The early proponents of educational technology were influenced by communications theory and were 
naturally drawn to issues regarding message design. In many ways, it makes sense to think of a particular message 
one wants to deliver and determine the appropriate medium to carry it. However, as Marshall McLuhan (1964) so 
famously pointed out, the medium is the message. In the recent history of educational technology, a constructivist 
philosophy has emerged as a valuable perspective on learning. 

In simple terms, constructivists believe that knowledge is constructed, not transmitted (Jonassen, Peck, 
Wilson, 1999). A constructivist perspective also emphasizes the elusiveness of meaning. To a constructivist, 
meaning does not exist outside of context. Learners construct meaning as individuals and as groups. The 
constructivist approach has provided a number of educational strategies. Learning by doing, cognitive 
apprenticeship, problem-based learning, and computer-based learning environments each embody key principles of 
constructivism (Duffy & Cunningham, 1996). The educator’s role focuses on guiding learners, not telling them. 
Educational materials are to be viewed as resources, not primary references. It may appear ironic, but the 
constructivist learning environment contains a multitude of messages that individually would each be suspect. 

The adoption of constructivism and the development of multimedia cannot be considered independent of 
each other. Multimedia can present multiple perspectives in a multitude of ways. Multimedia encourages 
constructivism, as constructivism provokes multimedia. Despite the fruitful marriage, text continues to serve as a 
primary mode of communication. Hypertext extends and bends meaning, but the words remain the same. We may 
think as constructivists, but continue to write as objectivists. 

General Semantics 

Fortunately, an entire field of study. General Semantics, addresses the meaning of words and texts from a 
post-objectivist point of view. The birth of this field of study is usually attributed to the publication of Alfred 
Korzybski’s “Science and Sanity” (1933). In this text, Korzybski describes the difficulties in using written language 
to refer to anything outside of other words. It also contains his oft quoted, rarely cited admonition that the map is not 
the territory. Constructivists can take his message to heart. If General Semantics and constructivism share similar 
philosophies, it follows that they also share tools and methodology. In this document, I provide thirteen guidelines 
for creating constructivist texts, along with a brief rationale for each guideline. 

Thirteen Guidelines for Creating Constructivist Texts 

1 . Focus on structure and relationships. 

General Semantics teaches us that words can mislead. Words are used to describe the world, but one must 
remember that words are not the world. Critics of this worldview see a sort of nihilism in this distrust of 
words. “Those persons who claim that knowledge is founded on collaboration rather than empirical 
science, or who claim that all truth is relative, are not instructional designers.” (Merrill et ah, 1996, p. 6). 
However, Korzybski (1933) explains that words in isolation lack meaning, but that words in context can 
convey meaning. To that end, he suggests that the relationships between words and the structure of words 
tell us meaningful things about the world we live. As such, constructivist texts must avoid the perceived 
need to define terms (e.g. “constructivism is...”) and instead focus on the relationships among words and 
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the structures they imply. A common dictionary illustrates this concept. Each word stands as an equal to all 
others and meaning emerges from the multiple interrelationships of words, not a reduction to smaller units 
or a dependence on first principles. 

2. Make context explicit. 

In an effort to reduce bias, textbook editors have limited the words or examples used in “acceptable” texts 
(Gladstone, 2003). As many critics have pointed out, this has only introduced a different kind of bias. Bias 
cannot be eliminated, because every text exists within a context. Rather than attempting to minimize the 
effect of context, writers should strive to make the context explicit. With explicit context, readers can more 
easily identify bias and potential problems with meaning. Korzybski notes that we are bound by time. We 
see the past through our own filters and those of others; the future cannot be known. We only experience 
the now. Our texts represent this. A map is not the territory, but it can illustrate the context in which it was 
made. 

3. Make the author(s) of texts explicit. 

To continue with the map metaphor, a map not only attempts to describe a territory, but also describes the 
biases and worldview of the mapmaker. Likewise, a text illustrates the worldview of the author. It is not 
possible to include all worldviews. And as many critics of “diversity” have pointed out, the desire to 
include all worldviews represents a particular worldview that is not universally shared. Again, one must not 
be reduced to a pessimistic outlook. By explicitly stating one’s worldview, the reader can compare and 
contrast that view with his or hers and react accordingly. For example, L. Summers (personal 
communication, October 8, 2003) used the qualitative technique of portraiture to describe the participants 
in a novel secondary education program. Prior to introducing the students, she describes her own person 
experiences as a young learner. The reader can then read her descriptions with a better sense of how the 
author’s experiences have shaped those views. 

4. Use quotes to delineate “questionable” concepts. 

Ironically, the practice in General Semantics of explicitly quoting individual words to draw attention to 
their dubious meanings has become a widespread practice among those that may have no knowledge of the 
field. Whether quoting in text or indicating quotes in the air during verbal communication, the gesture can 
be used and understood to indicate that the quoted word or phrase may not indicate what it purports to 
indicate. A challenge for authors is to increase its practice in “formal” texts. 

5. Use “et cetera” and other devices to indicate the limitation of a particular description. 

Another easily understood mechanism is the use of “et cetera” or “etc.” to indicate that a finite list is not an 
all-inclusive list. From the perspective of General Semantics, the period as a form of punctuation can imply 
a finality that is not intended (Kodish & Kodish, 2001). When presenting examples, or other enumerations, 
the frequent use of “etc.” will remind the reader that the author has not completely mapped the territory 
described. 

6. Use multiple representations. 

Words provide one type of representation. Pictures provide another. Graphs provide a third. Korzybski 
(1933) was particularly drawn to graphics because of their ability to represent structure and relationships. 
Multiple representations provoke readers to compare and contrast the concepts illustrated. As such, a 
constructivist text would be best served by providing at least two, but possibly more representations of any 
topic. Although work in General Semantics appears to focus on text, any symbol system can cause the same 
types of problems as misleading text. Our current “comfort” with televised images and instant access to 
information has introduced many of the same misconceptions that we have only recently begun to 
appreciate in text. 

7. Use metaphors. 

Teachers and interface designers use metaphors to connect with learners. However, metaphors have been 
criticized as being too limited (for example. Nelson, 1990). From a General Semantics point of view, the 
importance of metaphors as abstractions is that they are obvious abstractions. In contrast, technical writing 
and “dry” text fool the reader in believing that they are “literal” descriptions of reality, without the 
nonsense of abstraction. Readers are generally comfortable with discarding metaphors when they have 
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outlived their usefulness. I would suggest that those same readers would often cling tenaciously to an 
abstract definition, despite the problems it may cause. 

8. Avoid absolutes. 

The nature of absolutes can be illustrated by the following example. An instructor produces a small empty 
jar and a larger jar filled with marbles. She then pours the marbles into the smaller jar until it is filled and 
asks the class if the jar is full. There may be some discussion where the top of the jar ends, but the students 
will eventually conclude that the jar is full. At that point, the instructor produces a jar of sand and 
effortlessly pours a large quantity of sand into the “full” jar. This can be done again with another jar of 
water. The point of the experiment is that the seemingly simple concepts of “full” or “empty” are bound by 
conditions and assumptions (the reader should note that it is not difficult to show that the “empty” jar at the 
beginning of the experiment was actually “full” of air and that despite being “full” more air could be forced 
in to the jar). Although the author should attempt to make conditions and assumptions explicit, avoiding the 
unnecessary use of absolutes will increase clarity. 

9. Avoid either-or formulations. 

Although we continually remind ourselves not to think in “black or white” or that a particular characteristic 
can be viewed on a continuum, either-or writing becomes a hard habit to break. A familiar example is 
“quantitative” versus “qualitative” research. The “natural” assumption is that quantitative research lacks 
quality and that qualitative research lacks quantity. If distinctions were necessary, it would be better to talk 
about quantitative-qualitative research and qualitative-quantitative research to indicate that both “types” 
contain similar attributes, but with different foci. 

10. Avoid the passive voice AND make the actors visible. 

Many writing instructors suggest avoiding the passive voice. General Semantics makes the suggestion with 
a very specific criticism: the passive voice hides the actors. Consider the following examples, “I wrote 
these guidelines after reading a number of sources on General Semantics.” compared with “These 
guidelines were written after the consultation of a number of sources on General Semantics.” In the first 
example, the author’s role in creating the guidelines is clear. The second example is more formal, the 
passive voice puts the guidelines first and the author’s role in their creation is minimized. 

1 1 . Avoid the “is” of identification. 

The problematic use of “is” has led some semanticists to suggest that it be eliminated entirely from the 
language (Kellogg, 1993). For our purposes, it should suffice to address some of the concerns. A first 
problem with “is” concerns misuse regarding identification. If a writer states “Bob is a liberal,” the reader 
is expected to believe that the world consists of “liberals” and “non-liberals” with individuals neatly 
classified into each. However, there is no litmus test to achieve this. If the writer proposes one, the 
classification becomes conditional upon that framework. This can be avoided by avoiding the “is” of 
identification. If the writer is concerned with “Bob’s voting record” or “Bob’s writings on economic 
theory,” it can be done without errors in identifying Bob. 

12. Avoid the “is” of predication. 

A related problem is the “is” of prediction (Chalip, 1969). By stating, “Joe is bad” a teacher has freed 
himself from higher-order abstraction. An analysis of that thinking would reveal “he believes that Joe is 
bad because he is disruptive in class.” He can then conclude that Joe is not “bad,” but Joe is engaging in 
behaviors that are disruptive to the class. From that perspective, a teacher can help Joe in reducing his 
disruptive behaviors rather than continuing with a false label. The distinction of individuals and behaviors 
has contributed to greater understanding in counseling situations. 

13. Provoke readers to question assumptions and question questions. 

Readers construct meaning. A constructivist text challenges the reader to not accept the words as “truth” as 
a matter of authority of the author. This does not mean that everything is permitted. As with any scientific 
endeavor, empirical or not, truth is provisional and subject to revision. Readers should be encouraged to 
test the conclusions of any text and expand upon it. Not only should the questions raised by the text be 
answered, but also the nature of the questions questioned. As the title of Korzybski’s (1933) text suggests, 
science AND sanity are required to further human knowledge. General Semantics does not replace science. 
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but rather supplements it with understanding about the nature of language. 



14. Et cetera. 



Conclusion 

Educators and instructional designers that are involved in the creation of texts would benefit from the work 
done in General Semantics. General Semantics is not a panacea (I leave as an exercise to the reader, the 
identification of the various errors I have stated that exist in this text). However, it can be taught to students through 
relatively simple activities (for example, see Maas, 2002). The practice of these guidelines is like a good habit. A 
little is good and leads to more crisp writing. When it becomes more natural, the result is more crisp thinking. 
Hopefully, constructivists will add these useful tools to their toolboxes. 

Those readers interested in learning more about General Semantics can begin with the following resources: 

• Any serious academic inquiry into General Semantics must begin with Alfred Korzybski’s 
“Science and Sanity” (1933). Its size (927 pages) and academic tone can be daunting to some 
readers, but it is the singular source from which the field emerged. 

• A less academic introduction to General Semantics can be found in “Drive Yourself Sane: Using 
the Uncommon Sense of General Semantics” by Drs. Susan and Bruce Kodish (2001). 

• The most well known journal devoted to General Semantics is “ETC: A Review of General 
Semantics” published quarterly by the International Society for General Semantics. 

• A wealth of information can also be found at the websites of the International Society for General 
Semantics (www.generalsemantics.org) and the Institute of General Semantics (www.general- 
semantics.org). 
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Abstract 

The Teaching and Learning Department at Virginia Tech has offered an online Instructional Technology 
Master’s Degree program (ITMA) since 1998. A 4-D online teaching program model is generalized from the 
perspective of an ITMA designer and grader based on her ITMA experience. The 4-D model consists of 1 ) First 
dimension-X Coordinate: Inside system. 2) Second dimension-Y Coordinate: Outside system. 3) Third dimension-Z 
Coordinate: Leadership and 4) Fourth dimension-Time: Continuous system. 

Introduction 

The Department of Teaching and Learning at Virginia Tech began to offer an online Instructional 
Technology Master’s Degree program (ITMA) in 1998. As a distance program, all the courses are offered online 
with the support of Blackboard and ITMA database system, accompanied with tutorials, textbooks, and CDs. With 
the success and growth of the program, the students have expanded from K-12 educators to corporate trainers and 
other educators outside the academic field. 

Since the courses were put online, the instructors had little involvement with the course delivery. A full 
time faculty member acts as a director and leads the whole program. Several doctoral students work under the 
director, responsible for the design, coordination and grading work. These students are at least in their second year 
doctoral study in Instructional Technology program. 

Most of the ITMA students are part time students. They take the online courses when they work on their 
jobs. For most of the courses, there are few required due dates for the assignments. Most of the due dates are 
suggested with the consideration of the distance learners’ characteristic - their time is not fully devoted to the 
learning. They need more freedom to control their own learning pace. 

Every registered student gets some free space on the filebox server offered by Virginia Tech. They can 
request more space if they need. They are required to put their ITMA assignments in their filebox and submit the 
assignments online. There is specific ITMA online student interface for the learners to submit their assignments. 
After the learners submit their assignments, the graders can access the submitted assignments via the graders login 
page and grade the assignments. The grades are automatically stored on the ITMA servers and posted online. The 
learners can access to their grades and graders’ feedback/comments via the web page with their user name and 
password. If they have questions concerning the assignment, the grades or some technical issues, they will contact 
the director, the coordinator or the graders via email or sometimes, by phone. 

Actively involved in ITMA program, we have lots of contact with the online learners and understand the 
ITMA internal working system and the external working system. The interaction and the experience help us to 
generalize a 4-D online teaching program model, which consists of 1) First dimension-X Coordinate: Inside system. 
2) Second dimension-Y Coordinate: Outside system. 3) Third dimension-Z Coordinate: Leadership and 4) Fourth 
dimension-Time: Continuous system. 
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Figure 1 . 4 - D Online Teaching Program Model 



4 - D Model 




First Dimension-X Coordinate: Inside System 

The inside system of an online teaching program is the basis of the program, which includes the personnel 
resources, the financial resources, the hardware and software resources, the intelligence resources, the internal 
technical and logistic support, the organization of the system and the cooperation and utilization of different internal 
resources. Without strong inside system, any online teaching program will not succeed, even though it can be 
launched at the very beginning. Take ITMA as an example, it has funding resources as the financial support; well- 
equipped computer lab offering good software and hardware environment; efficient leadership to guide the program; 
full-time faculty member and doctoral students working for the program and good cooperation among the personnel 
involved in the program. 

Second Dimension- Y Coordinate: Outside System 

The outside system is the environment of an online teaching program. A friendly environment will help to 
strengthen the program. On the other hand, a poor environment will definitely undermine the development of the 
program. The outside system includes the institutional cooperation, the social recognition of the institution, the 
general attitude towards the online teaching program, the student resource, external technical and financial support 
and so on. The outside system has different levels, for example, the international level, the national level, the 
regional level, the community level, the institutional level, the learner level, etc. 

Third Dimension-Z Coordinate: Leadership 

To keep running a successful online teaching program, leadership is the key. Good leadership will help to 
organize the teaching program, build up team cooperation, set up flexible policy and coordinate the internal and 
external system. Take ITMA as an example. With the increasing number of courses offered and expanding student 
body, ITMA’s success comes from its effective leadership and the team cooperation developed by it. The 
department Chairperson believes in the future of the IMTA program and continues to support it. A full-time faculty 
member was appointed as director to manage the program. He devotes most of his time to the program, builds up the 
rapport relationship among the inside system and coordinates the relationship with the outside system. Everyone in 
ITMA has clear responsibility and reports to the director. The communication channel is smooth and the team 
members are empowered. Small problems are directly sent to and taken care by the right person who is responsible 
for them. Big concern is brought up at the weekly meeting and discussed together. The decision is made on the 
basis of the learners’ interest and the institutional requirement. All of these come from the leadership and contribute 
to the success of ITMA program. Without the effective leadership, nothing could be achieved. 

Fourth Dimension-Time: Continuous System 

An online teaching program is different from an online course. It is a systematic enterprise, which needs 
long-term investment and continuous effort. There is some kind of misunderstanding toward online teaching 
program. Someone simply takes it as a one-time behavior of wrapping the course contents into Blackboard, WebCT 
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or other software package, which is totally wrong. To initiate an online teaching program needs large amount of 
investment, such as hardware investment, software investment, personnel investment, technical support, etc. And to 
maintain the program needs even more efforts. Without the continuous effort and support, the investments will be 
largely wasted and there is no way to get the full out of the initial effort. 

The 4-D model is generalized from the successful experience of ITMA and also has a general application 
sense for other online teaching programs. With the continuation of ITMA practice, more and more experience will 
be generalized and contribute to the development of online teaching program model. 
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Abstract 

This paper will discuss the authors' experiences in developing and implementing an undergraduate and 
graduate program in Instructional Technology in the Department of Instructional Technology and 
Telecommunications (ITT) at Western Illinois University (WIU). The emphasis of this paper is to compare and 
contrast the unique nature of the undergraduate and graduate programs. 

Introduction 

Instructional design and technology skills are becoming essential in a variety of settings. The future of 
education, business, and industry rests in the hands of those who can use technology to provide the training 
necessary for continued growth and success. Nearly every organization, large and small, is engaged in ongoing 
instruction and training to maintain its competitive edge and to keep employees productive and up-to-date. As the 
need for training, continued education, and professional development grows, so does the need for expert personnel 
with Instructional Technology (IT) skills. 

Brief History and Background 

For many years WIU offered an undergraduate degree in Photography/Media. In 1996 it began offering its 
first Masters degree in Instructional Technology. In 1997 WIU began offering an undergraduate degree in 
Instructional Technology and Telecommunications. During the last five years both the undergraduate and graduate 
programs have grown dramatically. 

The ITT Department at WIU now has a total of 1 14 undergraduate students enrolled in its undergraduate 
program and 190 enrolled in its graduate level program. The majority of the undergraduates obtain jobs in the 
corporate world as web-designers, multimedia producers, performance specialists, and trainers. At the graduate 
level, the majority of our students are public school teachers seeking a Masters degree to provide them with 
technology skills to use in their classrooms and schools. To date, 130 students have graduated from our Masters 
program in ITT. The ITT Department maintains a very high placement rate of undergraduate and graduate students, 
with graduates holding positions at various levels of industry, government, and education. 

The ITT Undergraduate Program 

The ITT Department seeks to develop in undergraduate students the ability to apply new and existing 
technologies to solving educational problems in a wide variety of instructional settings including schools, business, 
industry, and government. Students accomplish the program's objectives by learning to design, develop, and 
evaluate effective, efficient, and appealing instruction, instructional materials, and information resources. The 
Bachelor of Science degree in ITT places an emphasis on providing students with both the technical skills, including 
training in the latest computer technology and applications, and the teamwork skills necessary to manage people and 
projects. Several of the students who decided to major in ITT were computer science majors that switched to ITT 
because they wanted to work with software applications (such as Director, Authorware, Photoshop, etc.) rather than 
program code. 

With an emphasis on hands-on experience, teamwork and real-world applications, WIU's ITT program is 
practical and flexible to support student's individual career goals. Most of the undergraduate classes in the program 
are project based. In most of the classes our students have to work in teams to complete instructional technology 
projects. All projects they complete become part of their portfolio, which is a requirement for graduation. 
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The ITT Graduate Program 

The ITT Department seeks to develop in master's degree students the ability to apply new and existing 
technologies to achieve educational goals in a wide variety of instructional settings including, but not limited to, 
schools, industry, government, heath-related agencies, and institutions of higher education. Students accomplish 
these objectives by learning to design, develop, and evaluate effective, efficient, and appealing instructional 
materials. 

In their course of study, students in ITT have opportunities to acquire conceptual and hands-on experience 
with effective teaching, learning, and collaborative design and development of professional instructional materials. 
In addition, internships, projects, and practicums provide students with chances to apply their skills in real-world 
settings both on and off campus. 



ITT Department Philosophy 

The ITT Department's conception of teaching and learning is based on a constructivist philosophy. 
According to constructivism, knowledge doesn't exist external to the learner. Rather, individual learners construct 
their own meanings based on their prior experiences. Learning is a result of construction, collaboration, reflection, 
and negotiation within a rich context in which learning is situated. Technology has the potential to support 
constructivist learning and be used for active, authentic, and cooperative activities. Knowledge building results when 
learners interact with their peers, collaborate, discuss their ideas, form arguments, and negotiate meaning. When 
used appropriately, technology provides a more decentralized environment where students take more control of the 
learning environment and become active constructors of knowledge while working on authentic tasks. The role of 
the teacher has shifted from knowledge transmitter to that of a facilitator who provides opportunities for interaction 
and meaning making to all learners. The use of technology and cultural tools to communicate, exchange 
information, and construct knowledge is fundamental in constructivism. Strategies for teaching and learning are not 
chosen to facilitate transfer of knowledge from the world to the learner's head, but to provide tools the learner will 
use to create meaning. 

Comparison of the Undergraduate and Graduate Programs 

Superficially, there seem to be quite a number of similarities between a graduate IT program and its 
undergraduate cousin. However, we have also found subtle differences. Following are some points of comparison 
that may help to highlight the obvious (and sometimes not so obvious) differences: 

Specific objectives and focus of each program. As described above, the graduate program is primarily 
focused on instructional design while the undergraduate program is focused on instructional development and 
production. 

Nature of the students in each program. The majority of graduate students come from educational settings. 
The majority of undergraduates plan to work in business and industry settings. Undergraduates are often more 
interested in developing instructional materials, they are less interested in designing instruction. They have also 
expressed preferences for production work, rather than project management. 

Establishing internship opportunities for students. Many of the organizations that offer Instructional 
Technology internships require students to be enrolled in a Master's degree program. There seems to be a clear bias 
among some instructional technologists responsible for administering company internship programs that students 
need to be working on a Masters degree in instructional technology to be qualified for an internship opportunity. It 
is more difficult for undergraduates to obtain an internship experience. 

Obtaining job interviews. Some companies make it very clear that job qualifications require a minimum of 
a Masters degree. Many organizations are not aware that undergraduate programs in IT exist. Its relatively easy for 
graduate students to obtain that elusive job interview. Undergraduates, on the other hand, seem to have a bit of an 
uphill battle. In addition, because undergraduates are often are production oriented, they sometimes interview for 
positions outside of the IT field. 

Student interest in developing good instruction. Because most graduate students come from an educational 
background, they are very interested in learning methods and techniques for developing 'good instruction.' In 
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contrast, some undergradute students are not really interested in developing effective, efficient, and appealing 
instruction, but rather are more interested in working with specific software applications on the computer. 

Benchmarking. It is difficult to do any benchmarking on the undergraduate program because there are so 
few programs available at an undergraduate level. While its relatively easy to compare the graduate program to 
other similar programs, the undergraduate program has few resources available to determine what should be 
expected or what should be considered standard. 

Keeping software and equipment up-to-date. Since the undergrade program is primarily focused on 
instructional development and production and courses are primarily project-based, undergraduates expect to use the 
latest and most effecient software/hardware. They seem to recognize implicitly that the potential jobs are directly 
tied to staying current with the latest software versions. Some graduate students feel the same way. (Usually those 
who are more production oriented or business/training types.) However, the majority of graduate students 
(especially those with education backgrounds) do not seem to demand the 'latest and greatest' software. It is difficult 
to keep current software applications and up-to-date hardware available. 

Does an undergraduate program work? 

Recently, Gustafson (2001) contacted many major instructional design and technology programs in the 
U.S. and asked them about establishing undergraduate IT degree programs. He found responses to such questions 
were almost universally negative. Evidently, faculty in many IT programs have very little desire to establish an 
undergraduate IT degree. He listed several reason for their reluctance. We thought it would be instructive to use 
these objections as a springboard for discussion. 

Lack of resources Lack of resources to expand programs is an important issue at most universities. 

Recently many states have initiated a series of budget cuts that impact their universities' bottom line. A new degree 
program can often be seen as a real resource drain, in terms of additional faculty, expanded office, classroom, and 
laboratory space, additional equipment, and operating supplies. At WIU, these issues were addressed and it was 
realized that if allowed to grow at the right pace, an undergraduate program could be an efficient enhancement to the 
graduate program. For example, laboratory space is more efficiently used, and faculty have access to undergraduate 
talent who's production skills are often remarkably good. In fact, the undergraduate talent pool has been an added 
resource not previously available to the College of Education and Human Services (the College within which the 
ITT Department is located). 

Belief that it would result in extra work Gustafson (2001) found that many faculty believe that 
undergraduate programs are more labor-intensive than graduate programs. Experience shows that it often takes 
intensive advising in order to keep undergraduates on track for graduation. And undergraduate degree programs 
must be carefully integrated with other departments in order ensure that general graduation requirements are met. 
Our experience has bourn this out. It's been quite difficult to keep courses scheduled so that undergraduates can get 
the courses they need in a timely manner. Also undergraduates simply do not display the level of emotional and 
intellectual maturity that graduates do. Intense advising is often necessary. 

Undergraduate credit hours are worth less Some institutions give higher value for teaching graduate 
classes. For example, student credit hour production may be calculated differently, with graduate classes receiving a 
much higher weight. Consequently, an undergraduate program may need to have many more students enrolled in 
classes, just to make up for student credit hours. At WIU, there are slight differences in how graduate classes are 
valued against their undergraduate counterparts. However, these differences are negligible. 

No incentives from the university Faculty may be reluctant to embrace an undergraduate IT program when 
they feel there is no incentive from the university for doing so. Some faculty may feel that an undergraduate 
program is less prestigious. Others may feel that they should be compensated at a higher rate since undergraduates 
require extra work. WIU's ITT program is structured so that faculty teach both graduate and undergraduate courses. 
Also, the University and College of Education and Human Services fully support the instructional technology 
program. This provides quite a bit of incentive. 

Belief that no undergraduates would enroll It might seem that students would be reluctant to enroll in an 
undergraduate IT program. However, our experience has been quite the opposite. The growth of undergraduate 
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program at WIU has far exceeded initial projections. Students from all different disciplines seek out and enroll in 
the ITT program. Undergraduates exhibit a very high degree of interest in multimedia and technology. 

Belief that there are no jobs for program graduates Some companies make it very clear that job 
qualifications require a minimum of a Masters IT degree. In addition, many organizations are not aware that 
undergraduate programs in IT even exist. As Gustafson (2001) pointed out, this is a real supply and demand 
conundrum. If there is little to no supply of graduates, its hard to raise awareness in potential employers. Despite 
this concern, graduates of the ITT program have done remarkably well in finding jobs. Since undergraduates are 
often are more production oriented, many find positions outside of the traditional IT field. However, they still end 
up using the technology skills they learned in ITT. And, as an added bonus we often get feedback from graduates 
out in the 'real world' who tell us how relevant their training is to their current job. 

Concern for possible negative impact on existing graduate programs Some faculty might be reluctant to 
embrace an undergraduate IT program because it may negatively impact a well established masters program. For 
example, if the knowledge and skills currently taught in masters programs were now being offered to 
undergraduates, then the masters programs would need to be modified (so that they would not be redundant). Our 
experience shows that there is some truth to this uneasy feeling. There are occasions, when our undergraduates have 
had a hard time seeing the difference between their coursework and that of the graduate program. On other 
occasions, employers of our undergraduates find it hard to believe that they only have a B.S. degree. 

Conclusion 

While on the surface there are many similarities between a WIU's graduate and undergraduate programs in 
ITT, there are still many differences as well. Both programs strive to give students opportunities to acquire 
experience with effective teaching, learning, and collaborative design and development of professional instructional 
materials. Internships, projects, and practicums provide both graduate and undergraduate students with chances to 
apply their skills in real-world settings both on and off campus. 

There are however, important differences in both the demographics of the two student populations, and in 
the objectives of the two programs. In terms of students, the undergraduate degree appeals to more production 
oriented, technology savvy students. The graduate program, on the other hand, generally appeals to students 
involved in public education and/or business training. This is probably due to the fact that, the two programs have 
slightly different orientations. As described above, the graduate program is primarily focused on instructional 
design while the undergraduate program is focused on instructional development and production. 

We believe that the conditions are favorable for establishing undergraduate programs in instructional 
technology. First, nearly all organizations, large or small, are engaged in the business of instruction and training. 
Second, in this information age, business organizations simply must address training to maintain their competitive 
edge and to keep employees productive and up-to-date. As Gustafson (2001) stated, "The current and projected 
future demand by business and industry for skilled instructional designers far exceeds the current and projected 
supply" (p.61). As the need for training, and professional development grows, so does the need for individuals with 
instructional technology skills. We feel that undergraduate programs in IT will help fill the gap and provide 
program graduates with the knowledge, skills and abilities to serve this growing need. 
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Abstract 

A collaborative project was implemented to provide project management experience for a group of 
graduate students in an online project management class and undergraduates in a face-to-face project management 
class. The graduate students role-played as instructional developers of a project for a client. The undergraduate 
served as subcontractors to the graduate students. The focus of this study was to explore the development of a 
community of practice across an online class and a face-to-face class. 

An undergraduate and a graduate class on instructional project management were offered during the same 
semester by the Instructional Technology Department at a U.S. state university. The graduate class was an online 
class; the undergraduate course was conducted in a face-to-face classroom, with Web-based support via 
BlackBoard. 

A simulation was developed where the undergraduates represented the Web developers and the graduate 
students represented clients who were working on a web redesign. Undergraduates worked first as competing teams 
to submit proposals. Once the winning proposal was chosen, the undergraduate worked as a team with subgroups to 
work on the project. Feedback was collected on the simulation from the undergraduate and graduate students. 

This paper presents the results of this study and the lessons learned. The critical issues will include the role 
of authenticity in developing a community of practice, the function of role-playing simulations in instructional 
project management training, and collaboration across online and traditional classes and across graduate and 
undergraduate classes. 



Introduction to Study 

Instructional project management is a complex discipline requiring a broad range of skills. Today’s 
workplace is increasingly requiring instructional project management skills of instructional designers. Evolving 
technology is causing the project elements to be managed to be even more plentiful, frequent, and diverse. 
Communication challenges become more demanding as management occurs over distances rather than at one 
location (Litchfield & Keller, 2002). Effective communication is at the heart of skillful instructional project 
management. 

Devising methods to teach these important skills effectively can be a challenge. It is essential for later 
transfer to job performance that the job context be integrated into learning (Clark & Mayer, 2003). Eor this reason, 
the authors chose to provide their students with the same experiences that they would encounter in the working 
world, supplemented with group interaction and coaching. 

The Instructional Technology and Telecommunications (ITT) Department at Western Illinois University 
offers an undergraduate B.S. program and a graduate M.S. program. In response to an increase in students who are 
site-bound or working full-time, ITT has begun shifting its traditional graduate classes to online classes. About 80% 
of the graduate courses are now offered online. 

This presents a problem for facilitating collaboration and mentoring between undergraduate and graduate 
students. In the past, graduate students could come into the undergraduate classes and work directly with the 
undergraduates. However, this was mostly handled by graduate and teaching assistants. 

In Spring of 2003, the graduate instructional project management course (ITT565) was converted from a 
E2E classroom to an online course. At the same time, the corresponding undergraduate course (ITT 330) was being 
offered in an electronic classroom that offered videoconferencing. Due to the cost of videoconferencing, this method 
of communication could not be offered very frequently. 

Nevertheless, it was felt that this situation offered a unique opportunity for trying a model for online 
collaboration between the two groups. 

The graduate students role-played as instructional developers of a project for a client. The undergraduates 
served as subcontractors to the graduate students. 
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A team of students from the graduate class developed a request for proposal (RFP) for the development of 
media assets for an instructional website. Teams of undergraduates responded to the RFP with their own proposals. 
The graduate team selected the best undergraduate proposal. All of the face-to-face undergraduate class worked on 
completion of the winning project. 

The communication process used face-to-face planning sessions, teleconferences, synchronous online 
discussions, and asynchronous discussions. Undergraduate students were able to play various roles in the project 
management process. Both the undergraduate teams and the graduate team were able to gain experience in 
negotiating with a client. The graduate students developed communication skills for managing projects when the 
various stakeholders are at a distance and telecommunication is essential. 

Background on Collaboration 

The cognitive apprenticeship model (Collins, Brown, & Newman, 1989) has been used in online mentoring 
models (Dorneich & Jones, 1997) to encourage the development of mentoring online. Numerous studies have 
documented the use of graduate students for the mentoring of undergraduate students (Russell & Daugherty, 2001; 
Koals, 2000). Similar models have been used in the context of having in-service teachers mentor preserves teachers 
via the Internet (Seabrooks, Kenney, & LaMontagne, 2000). 

A community of practice can be somewhat difficult to develop and maintain in online classes and even 
traditional classes (Dorneich & Jones, 1997). This is especially true with hybrid configurations where part of the 
students meet face-to-face and other students are online and meeting in asynchronous discussion groups. 

Another challenge is trying to develop a cognitive apprenticeship program across graduate and 
undergraduate students (Russell & Daugherty, 2001; Koals, 2001). This process involves finding a common project 
or projects with appropriate levels of mastery for the students (Clark & Mayer, ibid). This paper will present a study 
that addressed both of these issues. 



Description of Study Design 

An undergraduate and a graduate class on instructional project management were offered during the same 
semester by the Instructional Technology and Telecommunications (ITT) Department at Western Illinois University. 
The graduate class was an online class authored on BlackBoard; the undergraduate course was conducted in a face- 
to-face classroom, with Web-based support via BlackBoard. 

The undergraduate class consisted of younger students in their junior and senior years of study. They had 
considerable experience in multimedia and online production, but little experience in real-world project 
management. The graduate students were considerably older, and most had at least several years of work 
experience. They had more training in instructional design and project management than the undergraduates, but 
somewhat less experience in most cases in multimedia production. 

The instructor of the undergraduate class was the on-campus chair of the ITT department. The adjunct 
instructor of the graduate class was an experienced project manager of instructional and multimedia projects, a 
doctoral student, and remotely located. It was not feasible for her to come to campus or meet her students face-to- 
face, most of who lived at a distance from the campus. 

It was decided by the two instructors of these classes to see: 

• if a community of practice could be developed across a hybrid instructional delivery system and 

• if a cognitive apprenticeship model could be used with graduate and undergraduate students. 

A team of graduate students role-played as instructional developers of a project for a client, the ITT department. 

The undergraduates served as subcontractors to the graduate students. 

A team of students from the graduate class developed a request for proposal (RFP) for the development of 
the university departmental web site. Teams of undergraduates responded to the RFP with their own proposals. The 
graduate team selected the best undergraduate proposal. All of the face-to-face undergraduate class worked on 
completion of the winning project. 

The communication process used face-to-face planning sessions, teleconferences, synchronous online 
discussions, and asynchronous discussions, both within and between the graduate and undergraduate teams. 
Undergraduate students were able to play various roles in the project management process. Both the undergraduate 
teams and the graduate team were able to gain experience in negotiating with a client. The graduate students 
developed communication skills for managing projects when the various stakeholders are at a distance and 
telecommunication is essential. 

At the end of the collaboration the students were asked to evaluate their experiences. Undergraduate 
students in the face-to-face class were administered an anonymous questionnaire addressing the following topics: 
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1) What worked well in this collaborative effort between ITT 330 and ITT 565? 

2) What could be improved on for this collaborative approach the next time? 

3) How real did the project feel in terms of working for an actual client? 

4) Did you feel that you were part of a larger learning community because of your involvement in this joint 
project? 

5) What did the graduate student do to contribute to the sense of community? 

6) What more could the graduate student have done? 

7) Did you feel like you were part of a learning community during this project? 

8) Did you perceive of the graduate student as a mentor or more experienced participant in this project? Why 
or why not? 

The participating graduate student wrote a submission for a class threaded discussion on “what worked and what 
didn’t” about his experience project managing the development of the instructional web site. He chose to use it as 
his evaluation rather than filling out an additional questionnaire. 

Description of undergraduate task 

The ITT 330 students were split into four teams. These teams responded to the RFP developed by the ITT 
565 clients. Each team nominated a project manager. 

A videoconference was setup for the four ITT 330 teams to present their proposals to the ITT 565 clients. 
The client and the two instructors rated the written proposals and presentation using a common rubric. 

Once the winning proposal was selected, four new teams were selected by the ITT 330 students themselves: 

Graphics 

HTML 

Content 

Editing 

Each team elected a team leader. An overall project manager was also elected by voting for entire project. 
This manager met with the team leader to form a project management team. This team was the primary point of 
contact with the client. 



Description of the Online Graduate Task 

Since the intent of this course was to provide a forum to learn about managing instructional projects, ITT 565 
provided students with guided real world experience as instructional project managers. These experiences took a 
variety of forms ranging from developing a statewide K-12 online teaching resources portal to producing 
instructional web sites for college courses, college departments, nonprofit organizations and even a department of 
the United Nations. Other students developed infrastructure for K-12 technology integration within a school system 
and “smart classroom” acquisition within a college. The student team who collaborated with ITT 330 was charged 
with working as project managers with a contracted production agency from the undergraduate class to complete the 
project of redesigning the university ITT department web site. They had to engage with the client to conduct a 
needs analysis and generate commercial, creative, technical and content requirements for the web site. They 
generated an REP based on these requirements and made their selection of an ITT330 team to do the work based on 
their judgment of how well they would conduct the work. They had to identify risks and develop a communication 
plan and management plan for working with the team. Once they had chosen the “agency”, they had to use 
continuous and effective communication with the agency and client to create a successful project. The required 
deliverables were: 
a project brief 
a method plan 
an REP 
final project 

with management elements provided by the graduate online students including: 

• a revised RFP 

• budget 

• schedule 

• instructional delivery plan® 

• look and feel 

• content outline 
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with media assets to be produced by the face-to-face undergraduate students with the oversight of the 
graduate students including; 

• graphics templates 

• story boards (as applicable) 

• site map 

• a fairly well realized final product, at minimum a paper or computer prototype 

Description of the Graduate Online Class Simulation 

The collaboration team began as two students. One was a Master’s student employed as a graphics designer 
and web developer at the main university campus, with extensive experience in client relations and managing self- 
contained projects (in which she managed her own work but no one else’s). The other Master’s student was a 
technical support employee at a remotely located university campus. He was a technical assistant lacking project 
and management experience. The team did a very successful job of collaborating among themselves, with the client 
and with the undergraduate students to complete a thorough needs analysis and develop an excellent project brief, 
method plan and RFP. At the completion of the RFP, the more experienced student dropped the class - and thus the 
project - citing overextended time commitments. The remaining team member met several times by teleconference 
with the undergraduate class and the class instructor to consult about the RFP, select the “agency” to be awarded the 
contract (in conjunction with the instructors), and to develop a schedule for project completion. An agency project 
manager was chosen. The production agency agreed to establish a production web site for the posting of work as it 
was completed. The production timeline for completing the work was limited due to semester constraints. 

After the production process commenced, work flowed in to the project manager but did not meet its 
schedule. The graduate online project manager kept in regular touch with his instructor expressing his frustration 
about lapsed deadlines and unreturned emails. His instructor coached him to be more proactive; she suggested that 
he (1) establish who was responsible for the content outline and (2) establish regular status meetings via phone call 
at least once weekly. The student did not implement these suggestions. At the end of the semester, a small portion 
of the media assets had been implemented, and there was no evidence that the content had been or written or even 
thoroughly planned for. The final project assignments that were exclusively management-related were completed. 

Outcomes for Undergraduate Class 

The response to the RFP proceeded very well. There was some frustration in a couple of team members 
who felt some of the members had not contributed as much or had missed deadlines. Team members were asked to 
rate their contribution and effort in the team and to rate the other group members. This information was shared only 
with the ITT 330 instructor. There was a fairly good triangulation of estimate effort by group members, including 
self-estimates. 

As the project shifted into work on the actual project, stress began to increase. Part of this was due to the 
scope of work and the timeline for the project. It also stemmed from the inherent group interdependence. Project 
meeting were held to allow different teams to work out conflicts within their groups and between groups. This 
corresponded with class and text discussion on conflict management. 

The ITT 330 project team turned in revisions for the web site, including new graphics, revised site map, 
new content, and a revised homepage. They also completed an individual survey about the simulation. 

The ITT 330 students worked very intently on completion of their assignments. During a relatively short 
period of time, the students came up with an outline and template for a redesigned website. 

Most of the communications occurred within the groups. After the initial two videoconference sessions, 
there was some email communication between the graduate student and the undergraduates. Most of the 
communication was between the project managers for ITT 330 and the ITT 565 graduate student. 

The ITT 330 students felt fairly rushed by the timeline. Part of the simulation was to learn to negotiate 
timelines with the client. They corresponded with the student-client and negotiated additional time and reduced 
tasks. 

There were also challenges within the groups. Some of the groups found themselves held up waiting for 
other groups to finish their work. Team leaders negotiated with other leaders to reschedule tasks and to get updates 
on deliverables. 

Collaboration was not perfect. There was limited communication with the graduate student. The amount of 
collaboration between ITT 330 groups varied. Some seemed to work closely through their team leaders. Others 
seemed to work independently. 
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While such collaboration is possible, it requires more structure than may occur in authentic settings. Many project 
participants stated that an authentic environment was created but expressed the need for more communication 
opportunities. 

A final foot note to the undergraduate project. Four of the students participated in an internship project 
working as a team with a real client. The project was a tremendous success, with the client offering to give 
recommendations to all four students. He also said the quality of the video project was equivalent to projects 
developed earlier by commercial developers. The students felt the experiences in the simulation helped prepare them 
for dealing with a client and how to function as team members in a instructional design project. 

Outcomes for the Graduate Online Class 

This collaboration, though it was not successful in producing all of the assigned deliverables, was 
successful as a learning experience for the student and as an example for the other students in the class. It provided 
a contrast to the many class projects executed by students who successfully completed their deliverables, enabling 
the students to see that a multiplicity of factors need to work together for a project to come off successfully. The 
success of a project is by no means guaranteed; it often involves the project manager’s overcoming -as well as 
anticipating and avoiding - numerous obstacles. Some of the barriers in this case were the need to manage over 
distance, an unrealistic time frame for the completion of work, role confusion, and the sudden departure of the more 
skilled and experienced team member from a two-person team. 

In his evaluation of the project, the graduate student wrote that he felt that overall the project had been a 
good experience. He cited collaborating with his partner to create the RFP and the process of working with the other 
class and class instructor to consult and evaluate the proposals by the vying “agencies” as positive elements. He 
also praised several administrative elements, such as the way the other class was broken down into design teams and 
the fact that there was a design agency project manager for him to interact with. However, he wrote that he felt that 
more communication and negotiation were needed between the two. 

Negatively, he wrote that he felt that the undergraduate instructor had too much control over the project in 
general and that this undermined his authority. He did not have the input he would have desired into the design, for 
the way in which the rest of the undergraduate class was broken up into design teams after the “agency” was 
selected, and for selection of the agency itself (this decision was shared with the two instructors). 

Threaded discussions, in which the students exchanged news and support about their projects, played an 
important role in the online graduate class. The communication between the participants was one of the class’ most 
successful elements in the estimation of both the instructor and the students. In the threaded discussion, one of his 
fellow students wrote in response to his report of his experience: “Earlier in the class we had a discussion regarding 
the democratic approach to projects. Sometimes I think a democratic approach can lose focus. I think your problem 
is you had too many chiefs and your authority as a project manager was not well defined at the beginning. Thank 
you for sharing that with us because I think that is a very important lesson.” Then she went on to ask him if there is 
anything he could have done to avoid the problem. 



Conclusions 

The researchers conclude that it is important to design future class collaborations to be more representative 
of reality, thereby avoiding role ambiguity. The project manager from the graduate online class was confused by the 
role of the undergraduate instructor and departmental chairman who simultaneously played the role of client and 
head of the production agency. Designing the simulation so that the undergraduate instructor’s role was clear cut 
would have helped the project manager, according to his evaluation, to feel more comfortable in exercising his 
authority. Thus lessons were learned in regard to the need for clarity in the development of simulations and the role 
of authenticity in the development of a community of practice. 

The need to allot enough time for production was another lesson. In the future, the authors will allow for 
more time in the production schedule as they plan. 

Regular effective communication is critical for all instructional project management, and its lack in this 
collaboration is something that all of the participants noted. In a face-to-face situation, communication is easier to 
accomplish. Special affordances, such as regularly scheduled meetings by phone or teleconference, need to be put 
into place to facilitate communication between remotely located members of work teams. Such meetings enhance 
the ability to negotiate and make decisions and are necessary for team member accountability and for the project 
manager to exercise the necessary authority. Designing such communication opportunities as routine meetings via 
phone or teleconference into the structure of the collaboration would have helped the process immensely. 
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The authors note that while communication between the two classes was flawed, communication within the 
classes was open and abundant. Routine structures were designed into the online class to facilitate communication 
between remotely located people. Thus even difficult project management experiences became object lessons as the 
class shared them via threaded discussion. The authors conclude that role-play simulations can function very 
successfully in teaching instructional project management, whether or not they are successful as projects. 

Implications for Future Collaboration Efforts 

There are several implications that can be drawn from this study. First, it is possible for collaboration to 
occur between an online class and a traditional classroom, by making use of computer-mediated communication. 
However, email may provide limited structure for such communications. Currently, we are engaged in another 
collaborative effort to use an online course to provide a collaborative environment for interaction between 
undergraduates and graduate students. This project is using a third BlackBoard area to provide collaborative space 
for undergraduates in a traditional class to interact with graduate students in an online course. 

Second, a hybrid project management simulation can be a useful tool for preparing undergraduates for: 

1 ) working in developing teams, 

2) learning how to interact and negotiate with clients, 

3) understanding how to write a proposal in response to an RFP, 

4) develop a presentation that may be delivered at a distance, 

5) how to communicate within project teams, and 

6) how to communicate with clients using various forms of communication. 

While using an in-class simulation to teach instructional project management is not unique, the hybrid use 
of graduate student in an online course with undergraduate in a traditional course presented some unique 
opportunities and challenges. As more instructional technology departments begin to offer undergraduate degrees in 
instructional design and technology, such a model demonstrates a way that undergraduates and graduate students 
can be brought together in a community of practice. The ITT Department at WIU will continue efforts to explore 
this hybrid model to offer cognitive apprenticeship and mentoring opportunities for our students. 

Finally, not all the results were entirely positive from this study and valuable lessons were learned. It is best 
not to mix metaphors in simulations. The graduate and undergraduate students felt some confusion because of the 
mixing of client and mentor role for the graduate student. Also, communications between the two groups may need 
to be very structured at first. Although, undergraduates and the graduate student were frequently encouraged to 
communicate with each other, both groups complained that there was not enough communication from the other 
group. 

This simulation is being repeated in the near future with the same classes, but with some changes: 

1) The graduate student will be in a project manager role, not a client role. This will allow graduate 
students to mentor undergraduates without having to “step out of character” for a client. It will also 
allow for more direct involvement in the project on the part of the graduate mentors without having to 
worry about stepping out of the character of the client. 

2) A fictitious client will be used for negotiation and feedback purposes only. Mentoring will be through 
the instructors and the graduate students. 

3) Communications will be strongly encouraged within the class members and across the classes. While 
videoconferencing will be used, most communications will be through a collaborative online course as 
vehicle for shared communication. Also, more frequent synchronous meetings will be scheduled 

4) The timeline for the project will be started earlier. 
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Abstract 

Opportunities for non-native English speakers to learn English in naturalistic settings are scarce. 
Computer-mediated communication ( CMC) tools have shown some promise in this regard, but existing online 
language acquisition tools provide learners with little opportunity to converse verbally with native speakers in 
natural contexts. The Electronic Language Acquisition Bridge (ELAB) model described here portrays an as-yet 
undeveloped language instruction system that supports incremental skill development within a single CMC 
environment. The implications of this approach extend beyond English as a second language (ESL) or English as a 
foreign language (EEL) instruction. 



Introduction 

Opportunities for non-native English speakers to comfortably learn English in naturalistic settings are 
scarce. Consequently, computer-mediated communication (CMC) has shown much promise in creating learning 
environments that are more egalitarian and less threatening than face-to-face exchanges with native English speakers 
(Chun, 1998; Warschauer 1997). Eurthermore, CMC offers the ability to design more structured and purposeful 
learning activities that the transient nature of face-to-face communication does not readily allow. CMC technology 
allows participants to capture their textual and verbal communication, enabling subsequent revision by oneself, 
peers and instructors. The ability to append feedback and annotations directly to learner dialogue makes feedback 
especially instructive and helpful. 

The goal of our project was to design a system addressing the continuum of technology-facilitated 
communication intended to generate incremental skill development in ESL/EEL. In addition, we attempted to create 
an environment for ESL/EEL learners to develop their language skills in the context of generating authentic text and 
voice -based exchanges. Collateral benefits included 1) opportunity for increased volume of communication; 2) 
practice not limited by class time and location (time & space). Consequently, we named our design ELAB- an 
acronym for Electronic Language Acquisition Bridge. 



Literature Review 

The rapid growth of information and communication technologies in recent years is associated with a 
correspondent increasing interest in computer-mediated communication. Since the late 1980s, the trend has been to 
integrate the possibilities and complexities for language teaching and learning. 

CMC is an umbrella term which refers to human communication via computers- transmit, store, and receive 
information by the users (Simpson, 2002). The structural format of CMC can be synchronous CMC, where 
interaction takes place in real time, and asynchronous CMC, where participants are not necessarily online 
simultaneously. Synchronous CMC includes various types of text-based online chat, computer, audio, and video 
conferencing. Asynchronous CMC includes email, discussion forums, and mailing lists. In our ELAB design, it’s a 
continuum of moving from asynchronous mode of CMC to synchronous one. 

CMC is a possible cognitive amplifier that can encourage both reflection and interaction because speech 
and writing have merged into a single medium. This kind of text-based communication has the advantages of; 1) 
allowing students to practice rapid interaction; 2) allowing students to have control for their space in the midst of 
interaction; 3) encouraging students to be more expressive in this mode than they are in the written mode alone; 4) 
stimulating creative thinking because of more preparation time allows (Warschauer 1997). In general, synchronous 
communication seems to encourage communicative fluency. Students’ output in the asynchronous communication 
was lengthy and more syntactically complex than that available in synchronous discussions. Students who 
communicate synchronously seemed to focus on meaning and disregard accuracy; students who communicate 
asynchronously had more time to plan their answers and monitor spelling and punctuation (Sotillo, 2000). 

Merron (1998) found that students using threaded discussions wrote more thoughtfully than students who 
were not given such opportunities. Chong (1998) repotted the interactive and collaborative nature of asynchronous 
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technology allows students to share perspectives and experiences, and to seek assistance. Further, threaded 
discussion “allows everyone to be heard” including students who do not normally participate in face-to-face 
discussions (Kamhi-Stein, 2000). 

Many computer-assisted language learning (CALL) studies indicated that network-based communication 
may be beneficial for enhancing learners’ interlanguage even more than oral conversations, as the learner can view 
their language as they produce it- monitor and edit their message (Warschauer 1997; Kitade 2000). One advantage 
of using threaded discussion for ESL/EFL learning is the delayed time allowed the learners more time to organize 
and develop their thought before they “speak out.” Kitade (2000) also pointed out that the lack of non-verbal cues 
[in non face-to-face and non-video environments] may facilitate negotiation of meaning as communicants must rely 
on verbal correspondence alone. However, the ultimate goal of CALL should be creating an environment that 
simulates the authenticity of face-to-face communication. Thus it was felt that a need exists to modify current text- 
based threaded discussions in order to provide ESL/EFL learners increased opportunities to improve both their 
written and spoken English skills. 

“Second language acquisition (SLA) theories advocate that oral interaction that requires negotiation of 
meaning is necessary for enhancing learners’ interlanguage” (Toyoda & Harrison 2002). The Interaction Hypothesis 
claims that resolving miscommunication (negotiation) enhances L2 learning, as it provides more opportunities for 
comprehensible input and modified output. The Output Hypothesis explains that producing output is one way of 
testing a hypothesis about comprehensibility or linguistic well-formedness and that learners’ hypothesis testing often 
invokes interaction between the learners and their interlocutor(s). Native speakers’ difficulties in following learners’ 
interlanguage may trigger feedback, which in return may induce changes in the learners’ output. Negotiation of 
meaning also occurs on occasions where the native speakers’ input is above the learner’s threshold level of 
understanding (Toyoda & Harrison 2002). 

“Whole language philosophy incorporates constructivism theory and proposes that, by experiencing whole 
written or oral discourse in meaningful units, students learn to analyze the parts and construct new knowledge by 
reordering or synthesizing relationships between the parts. Language acquisition is, therefore, an active process in 
which the students focus on cues and meaning and makes intelligent guesses (Steep-Creany, 2002).” 

Research by Mayer and Moreno (1998) has indicated that in certain contexts, audio is more effective than 
text as a means of conveying information. They argue that this is due to the dual processing theory of cognition- 
working memory has two channels for processing information, an auditory and a visual one. 

Our hypothesis is that when ESL/EEL students are presented with both modes (text and audio), their 
mastery of content as well as their communicative skills in English will be better than when they receive only one 
mode in isolation (traditional text-only threaded discussion). Based on our review of previous studies, we suggest 
that CMC -based language acquisition tools should emphasize the development of both speaking and listening 
comprehension skills in spoken English. A design for such a tool will be discussed in this paper. 
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ELAB- Design Description 

Our design proposition was based on the concept of building an electronic bridge that will enable EFL/ESL 
learners to become active and effective participants in exchanges with English native speakers through incremental 
skills development. Learners progress from asynchronous threaded text discussion to simple asynchronous threaded 
voice exchanges to full, synchronous voice-enabled conversation. We hoped learners would benefit from gaining 
more opportunities for increased volume of communication and practice with no limit in time and space. 

Figure 1. 
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Phase I: Asynchronous Text Communication 

Groups are formed of approximately 4 students, divided evenly between native and non-native English 
speakers with the addition of native-English speaking moderator. The moderator will provide frequent feedback in 
the form of reflective questions. Instructors may wish to have native English speakers respond verbally (w/audio) to 
non-native English speaker’s textual comments as an additional listening comprehension practice for EEL/ESL 
students. 

Phase II; Asynchronous, Delayed Audio Conversation 

Each student will record a verbal (audio) response for their peer as a feedback. The technology used here 
will also enable users to compare their audio file with their partner’s to compare such qualities as use of expression 
and comprehensiveness of the recorded message. It is noteworthy that communication in Phases I and II is 
asynchronous. Students will record comments, replay them, and can choose to rerecord any number of times before 
posting them. 
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Phase III: Synchronous, “Live” Audio Conversation 

As non-native English speakers become more comfortable with their performance on the threaded audio 
exchanges, they will engage in synchronous voice-based discussions via the web. Video technology was 
intentionally not designed into ELAB, but could be added assuming minimum technical requirements could be met. 

In order to verify our hypothesis for designing ELAB, we conducted some user testing. Both EEL 
instructors and students participated in the user testing. Both discussed the value of ELAB for large class situations. 
In such cases, ELAB elicits frequent discussion from all students in smaller groups and makes non-participation 
difficult. Both teachers and student felt that, depending on the nature of the activity that ELAB would be useful 
across all EEL competency levels, and could be used effectively in all main subject areas (Reading, Writing, 
Listening, Speaking, and Grammar). 

After review some of the previous studies, the benefits for ELAB we perceived are listed in the following 

bullets. 

• EEL students can engage in incremental skill development. 

• EEL students will have the opportunity to receive context-sensitive feedback based on text and speech- 
based elaboration. 

• Cost effectiveness of learning technology (threaded discussions; asynchronous and synchronous voice- 
based technology all over the Internet) 

• Sharing of knowledge & culture and making friends in a meaningful context 

• Leverages existing and proven technologies currently available on the market 

• All communication is captured for reflection and feedback 

We also perceived some potential challenges for designing an integrated system for multi modal online 
threaded discussions like ELAB (Electronic Language Acquisition Bridge) based on the previous studies and our 
results of user testing. 

Synchronous communication will require sensitivity to time zone differences. 

Asynchronous communication activity relies heavily on the user-friendly interface design and well-organized 
instruction guidelines for learners to follow. Thus the text design (verbal & visual) should be made as clear as 
possible. 

To create an environment that simulates the authenticity of face-to-face communication, yet provides ample 
opportunity for EEL learners to review and reflect on their contributions and those of their native English 
speaking peers. 

User satisfaction with ELAB is heavily tied to the relevance and quality of learning activities designed by 
instructors. Eor the product to be successful, care must be taken to design support structures within ELAB to 
guide and encourage quality activities. 

In an effort to reduce the anxiety that many non-native English speakers experience when communicating 
with native English speakers, ELAB revolves around incremental skill development as learners advance from text to 
voice-based elaboration on various topics. As such, EEL learners can build a greater sense of self-efficacy by 
gradually progressing from threaded discussion to simple voice message recording with peer feedback to 
synchronous voice-enabled web-based discussion. 

Research suggests the need for full spectrum, full service, multi-modal CMC tools. ELAB fits the needs of 
ESL/EEL teachers and learners and has its theoretical basis in language learning. User-tests showed that ELAB is 
best utilized where speech (listening and speaking) and conversation were a helpful and important part of the 
learning goals. 
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Simulations: An authentic learning strategy for HPT? 

Kathleen W. Ingram 
Chuck Collins 

University of South Alabama 

Abstract 

In an ejfort to provide an authentic learning experience, one that promotes critical thinking and problem- 
solving, I designed, developed and implemented a Performance Technology extended simulation for the masters and 
PhD students enrolled in a Performance Technology course. The simulation focused on challenges faced by 
performance technologists who work as internal consultants in business and industry. This presentation is a 
discussion of my experiences running an extended Performance Technology simulation, design considerations, the 
amount of research required to develop an extended simulation, student perceptions about their learning from the 
experience, and lessons learned. As an organization tool and because of the process it denotes, the paper is 
structured around the core elements of instructional development: analysis, design, development, implementation, 
and evaluation (ADDIE) (Gustafson and Branch, 1997). 

Analysis 

The Performance Context 

The information age, globalization and technology have brought changes to the complexity and speed at 
which we do business. In the complex working environment of the 21“‘ century, professionals need to be able to 
accomplish complicated tasks (Prusak 1997 in Oblinger and Verville 1998), which require a high level of knowledge 
and skills. One of the skills necessary for success in today’ s global and volatile business environment is the ability to 
solve problems that are both complex and ill-defined. However, Oblinger and Verville (1998) report that the 
American Council on Education (ACE) found that graduates of higher education “. . .lack the critical skills: listening, 
communicating, defining problems, leveraging the skills of others in teams, and functioning effectively in an 
ambiguous, complex and rapidly changing environment (ACE 1996, p. 8)”. 

Human performance technology (HPT) is an applied discipline whose goal is to improve human 
performance in the workplace (Stolovitch & Keeps, 1999). The practice of HPT involves the application of a set of 
methods and procedures, developing a strategy for solving problems, and for identifying and implementing 
opportunities related to performance (International Society for Performance Improvement, 2003). Stolovitch and 
Keeps state that human performance technologists “ . . . adopt a systems view of performance gaps, systematically 
analyze both gaps and systems and design cost-effective and cost efficient interventions that are based on analysis of 
data scientific knowledge and documented precedents. . .” (p. 10). Based on the definitions of HPT and the 
professionals who practice HPT it is easy to see how training for this field requires the same skills that Oblinger and 
Verville state are needed in higher education. 

The Learner 

Institutes of higher education are experiencing a large in-flux of what were once known as ‘non- traditional 
students’ due to the changes in business environment, which have necessitated that employees continuously improve 
their skills in order to stay current. In fact, Kasworm (2003) reports that in 2010 it is estimated that 38.2 percent of 
the collegiate population will be twenty-five years or older (p. 4). Kasworm (2003) defines an adult as someone who 
often has “... competing sets of adult roles reflecting work, family, community, and college student commitments 
(p. 3)”. This description of an adult learner in higher education matches the characteristics of the students in the 
graduate courses I teach. 

Learning Context 

As an instructor in higher education and as an instructional designer, I often struggle with providing my 
adult graduate students with authentic projects that provide them both opportunities to develop their critical thinking 
and problem-solving abilities as well as provide adequate assessment of their conceptual and procedural knowledge. 
Many instructional strategies for the promotion of complex thinking skills involve learning activities that take place 
in actual work environments. However, the compressed time period of a semester, the characteristics of these adult 
learners in the graduate program in which I teach, as well as the risks of “practicing” in a real situation where the 
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consequences are not foreseeable compelled me to look for an alternative to “real” projects for my graduate 
Performance Systems Technology course. 

The literature suggests that the use of case studies and simulations provide an authentic context that can 
promote critical thinking and help the learner .approach problem situations in the same ways that practicing 
professionals do” (Stepich, Ertmer, and Lane 2001, p. 53). In fact, case studies and simulations have been used 
extensively in higher education (e.g. Sanci, Day, Coffey, Patton & Bowes, 2002; Jennings, 2002; Hemmes, Long, & 
Rowland, 1998; Stepich, et ah, 2001; Stephen, Parente, & Brown, 2002) as well as in training in business (Linema & 
Makkonen, 2003; Aldrich, 2004) and military (Noble, 2002) environments for presenting real world situations that, 
due their nature are ill-defined and complex, require critical thinking in developing holist or systemic solutions. 

Simulations are experiential learning strategies that are used "... to help individuals or teams develop 
problem solving and decision making skills" (Van Tiem, Moseley & Dessinger, 2001, 60). Simulations as an 
instructional strategy in higher education are designed to help students acquire "...broad discipline-specific 
knowledge that they are able to later transfer into a professional setting " (Hertel & Millis, 2001, p. 1). In addition 
the participatory and experiential nature of simulations are thought to improve learner motivation (Laria, 2001) and 
deep learning (Hemmes, et ah, 1998). In addition, simulations provide a safe environment for learning, one that 
controls the variables so that the learner develops a deeper level of competence before participating in an actual 
performance environment (Treadwell & Grobler, 2001- p 477). 

Although I had utilized case studies in several of my classes, I had never used simulations as an 
instructional strategy. However, based on my initial learner, performance context, and learning context analysis, I 
determined that an extended (Hertel & Millis, 2001) simulation was a strong instructional strategy. My learners were 
nontraditional learners, i.e. they are older, work full time, and have children, and therefore would be hard pressed to 
conduct a full-fledged performance analysis in an actual work setting. It had been my experience, as both a learner 
and an instructor, that there were many drawbacks to requiring learners to apply new concepts in an actual setting 
and in "real time". The performance context, any company that requires a performance analysis, would require the 
flexibility of knowing how to apply knowledge and skills to ill-defined problems. Therefore the learning outcomes 
from an extended simulation would transfer well to the performance environment. 

Designing the Simulation 

Simulations, an experiential exercise appropriate for developing problem-solving skills (Gredler, 1996), are 
by definition and design a constructivist learning strategy. Dick (1991) in his review of constructivist learning 
strategies, as they relate to instructional design, raised a concern about placing students in a complex situation for 
which “Some students probably are capable of dealing with this level of complexity, but certainly some must be 
overwhelmed” (p.42). While my learner population consisted of adults, they were novices in the area of performance 
technology. Therefore I used a combination of instructional strategies- case studies, HPT model-building, design of 
interventions- early in the course in an effort to provide the necessary conceptual basis (entry level skills) for the 
simulation. 

Snow, Gehlen and Green (2002) compared two approaches to introducing a simulation in a graduate 
business course, a stand-alone approach and an integrated approach. While the factors of confidence and usefulness 
were found to be the most valuable in judging the success of the simulation, the authors also found that the way a 
simulation is introduced to the participants plays a role in their learning outcomes. The authors found that an 
integrated approach increased: 

■ The learners’ confidence in their ability to play the game 

■ The importance they placed in trying, and 

■ The effort they reported making (p. 531) 

Similar to many instructional strategies, educational simulations are designed around certain learning 
objectives (Hertel & Mills, 2002) or outcomes. However, experiential simulations are also a “...scenario of a 
complex task or problem that unfolds in part in response to learner actions” (Gredler, 1996, p. 523). Therefore, while 
the objectives, choice of business, and general simulation information were developed into an outline before the 
course began, the simulation actually emerged as the class progressed. It wasn’t until I saw what problems the 
students were having with the material, how the concepts were coming together, and through extensive research into 
current business trends that I was able to completely develop all of the simulation arifacts. 

While both games and simulations have common characteristics, e.g. student control of action, Gredler 
(1996) states that games and simulations differ in three important ways. The first difference is that games are 
competitive, and therefore the purpose is to win, whereas the purpose of a simulation is to have students experience 
the consequences of the decisions they’ve made while in their roles. Secondly, the event sequencing of a game is 
usually linear and a simulation the sequence is determined by the decisions the participants make at each decision 
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point. Finally, the consequences of an action taken by a student are determined by rules in a game situation, whereas 
in a simulation the consequence is a “dynamic set of relationships among several variables that (1) change over time 
and (2) reflect authentic causal process...” (p. 523). 

Much of the research literature on the use of simulations is primarily found in medicine, English as a 
second language (ESL), computer simulations or secondary school literature. While informative, this literature base 
was neither prescriptive nor broad-based enough to infer design principles for graduate students in the specific area 
of performance technology. Therefore, the design choices of the course were most influenced by Hertel & Millis’ 
(2002) book Using simulations to promote learning in higher education. Their experiences, while based on 
simulations in the field of law, within the context of a higher education classroom and the use of extended 
simulations fit the goals for an authentic learning experience that promotes active participation, problem-solving, 
and deep learning. 

In fact, in his analysis of research that had been presented at Associative for Business Simulation and 
Experimental Learning (ABSEL) conferences over the past 25 years Earia (2001) found that; 

■ students have more positive attitudes toward learning from business games (simulations) than from 
other teaching approaches 

■ greater instructor involvement improved student performance 

■ more cohesive teams performed better than less cohesive teams 

■ a positive attitude and commitment to the simulation improved performance 

■ teams in high simulation/game grade-weighted sections outperformed teams in lower grade-weighted 
sections 

My research on the challenges companies are currently facing involved reading current research, popular 
books, and surfing the Internet. Eor example, in her book The Company of the Future, Cairncross (2002) discusses 
the past, current and future uses of the Internet in business. She states that there are five aspects of Internet 
technology that make it a powerful tool for change (p. 4). She states that these aspects have brought about quicker 
adoption of the technology, increased globalization and innovation, fostered new decision-making processes, and 
sped up the innovation process. Really want to stress the similarities between the management issues she addresses 
and those addressed in HPT. These ideas fit well with the International Society for Performance Improvement’s 
(ISPI, 2003) definition of human performance technology, the processes it follows, and the intended outcomes of the 
process. 

Cairncross’ (2002) ideas related to the company of the future, specifically her emphasis on the future of 
Internet for business practices, not only provided a framework for the “what should be” for current practices for the 
company portrayed in the simulation, it also provided the impetus for choosing a company that was already using 
the Internet for some of it’s communications. 

Development & Implementation of Simulation 

As stated earlier, the design and develop of materials for the simulation evolved throughout the simulation 
experience in response to student needs and perceived learning opportunities. This simulation was not an individual 
effort but rather a team process. Most complex problems, such as performance technology problems, require the 
input of several perspectives. In fact, collaborative team-building skills are expected and utilized extensively in 
business and industry. 

Although there were two teams, the design was an experiential simulation (Gredler, 1996) rather than a 
game. Each team worked independently, received the same information, the same opportunities for data collection, 
and was graded on the same rubric. The only difference was the choices they made on what data to collect based on 
the questions they asked in the interviews. The answers to those questions, which were loosely scripted, influenced 
the teams’ decision-making process and their choices about which information to pursue. Eor example, in the 
debriefing the two groups had very different experiences in their interviews with two of the players. 

Member of Group 2: It’s very interesting because for our interviews, I felt like the VP of Marketing and the 
VP of Research and Development switched roles. The VP of Marketing was much more difficult to get 
information from... sharing information VP of Research and Development was, now, here’s what’s 
interesting, that one of your (referring to Group 1 ) major suggestions or interventions was relevant to 
research and development and ours was not; we felt like everything was on track, I mean isn’t that kind of 
interesting? 

The perceived performance problem for Eort Dodge Animal Health, a subsidiary of Wyeth 
Pharmaceuticals, was stated as a failure to meet sales expectations. This information was provided to the two “inside 
performance technology teams” during an interview with the “VP of Human Resources” of Port Dodge (the 
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instructor). During the same interview the teams were provided with a schedule of the stakeholders they would 
interview. The stakeholders consisted of upper and middle management. As the simulation evolved, and at the 
request of the teams, an interview with customer service representative was subsequently scheduled. There were a 
total of six interviews for each team. Then, from the stakeholder interviews, and other data collection activities 
outside of class (e.g. benchmarking) the teams developed their needs assessment and the processes for performance 
gap resolution. 



Simulation Experience from a Student Perspective 

The integrated strategies that we completed in the first half of the semester aided our recall prior knowledge 
as it relates to HPT and afforded us an overview of the steps necessary to perform a thorough performance analysis, 
thereby providing us with a well-rounded understanding of the challenges we would face during the simulation. 
These early exercises provided a basis for consideration of how to best develop appropriate questions that would 
facilitate data gathering process throughout our interviews. 

The simulation experience was quite realistic. The instructor used a real company and devised a authentic 
problem that challenged us to consider various conclusions and how we would implement them. Then, based on our 
initial hypothesis we gathered data through interviews with several “actors”. In addition, the benefit of using a real 
company allowed us to use the Internet to research similar companies and compare their practices to those of Fort 
Dodge for benchmarking purposes. Both of these practices increased the fidelity of the simulation by providing an 
interactive experience by exploring multiple perspectives and practices. 

Group processes that involve reflection and higher order cognitive considerations have proven to be 
instrumental throughout my work as a graduate student. By having the opportunity to share and scrutinize my ideas 
and share various perspectives with a group of students, I feel not only that my current learning experiences been 
enhanced but that I have also been stimulated to develop a philosophy of learning and a direction for future research. 
In addition team experiences have taught me to evaluate and form arguments about other viewpoints based on my 
prior leaning experience. 

Initially, we reviewed the data from the interviews we had conducted and developed our own conclusions 
before meeting. Next we met to discuss our conclusions and collaborate to reach consensus on what our next steps 
should be. This was a scaffolding process by which we built upon each phase of data gathering. 

Once all the interviews were complete we then took a step back to piece together our perspective of the 
performance gap and begin to establish a rationale for possible solutions. In addition, we used the Internet to 
research similar companies that we would use as benchmarks for a comparison of the Fort Dodge operations. We 
focused on Pfizer pharmaceutical because they have an animal health division, were number one in the nation for 
sales in the animal health market, and appeared to exemplify “best practice”. Through reading Pfizer’s practices and 
comparing them with what we had learned from our data collection on Fort Dodge we clarified the goals applicable 
to our performance analysis. 

The opportunity to explore, without the pressure of making mistakes, provided us with the latitude 
necessary to investigate the performance gap thoroughly and make recommendations without fear of negative 
consequences. In a real world environment there would be obstacles I would have to face as an adult graduate 
student, i.e. the availability of time, coordination of people and data gathering. These obstacles would have limited 
my ability to accumulate data for a successful learning outcome. 

For me, the data gathering and analysis process was the most beneficial aspect of the simulation; the 
opportunity to analyze information, and propose a solution was an excellent exercise in the application of 
fundamental of HPT. Give and take within our group was excellent and provided a forum for dialog, theorizing and 
solution proposals. 

As for any dislikes or changes I would made; I would have liked to see the simulation develop an Intranet 
as a repository of information for both the students and the actors to assure consistency and uniform availability of 
information. 

Did the simulation fit my needs as a learner? Yes, I found myself immersed in the experience, it opened a 
path to analyze circumstances and think about solutions. I was able to explore a multitude of variables affecting 
performance and thereby better understand the phenomenon of a performance gap analysis. I was able to appreciate 
the process of designing a model for a particular performance need and then formulate an implementation program 
for resolution of the performance solution. 
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Formative Evaluation 

The formative evaluation process included both group and individual feedback. The first part, a group 
debriefing session, was conducted after the simulation was completed and each group had presented their findings 
and recommendations for Fort Dodge Animal Health. Assessment for constructivist learning experiences is 
challenging. While the focus for a constructivist-learning environment is on facilitating the learner’s construction of 
knowledge, the intent for the instructional designer (instructor) is to determine what has been constructed (Dick, 
1991). Debriefing, a process for helping the learner to connect their experiences in the simulation to their past 
experiences, the learning objectives of the simulation, and future practice (Lederman, 1992), serves as an assessment 
tool for experiential learning activities (Hertel & Millis, 2002). 

The debriefing was on Hertel and Millis’ (2002) questions for conducting an extended simulation briefing. 
An example of one of the questions asked near the beginning of the debriefing was “Did your level of emotional 
involvement change during the simulation?’’ One student’s response was that “. . . as we talked to the interviewees 
and gather more information, I think it changed a little bit because you became more aware of what was going on in 
the company and . . . what was at stake.” The follow up question to the same student was “Would you say that you 
became more involved?” The student’s response was “More involved, absolutely, as you became more familiar with 
the situation.” 

Immediately after the debriefing, each student completed an individual questionnaire about the course, 
which included questions related to the entire course and specifically to the simulation. One of the questions asked 

the students to complete the statement “What I liked best about the course was because ” The majority of 

the students (4/5) stated that the simulation was the best part of the course. The reasons were because it was 
“realistic”, “a realistic application of skill”, and “application is the only way to understand effective intervention 
techniques”. The students were also asked to rank the three most relevant assignments in the class. Table 1, below, 
shows the results from that question; 



Table 1 Students’ rankings of course activities 



Rank 

( 1 =highest) 


Number of 
respondents 


Activity 


1 


5/5 


Simulation 


2 


2/5 


Simulation report (PT Analysis Report) 


2 


1/5 


Intervention Assignments 


2 


1/5 


Case studies 


3 


2 


Case studies 


3 


1 


Individual project 


3 


1 


Intervention Assignments 


3 


1 


No answer 



Finally, when asked how effective the simulation was for their learning the students gave a 5 pt rating (on a 
1-5 rating scale, with 5 as the highest) across the board. 

Student Evaluation 

I think the preparatory work that the instructor did was well thought out and enabled us to become quickly 
immersed in the simulation and learn as much as possible about a HPT environment without the risk of a real world 
application. The tools we were provided with were minimal, yet sufficient enough to facilitate a productive learning 
environment through the simulation. The opportunity to experiment in a learning environment that provides the tools 
for a successful performance evaluation, such as interviews, benchmarking and general information about the 
company we were to analyze, greatly enhanced my learning experience and the retention of material that I learned 
through the exercise. 

Lessons Learned 

While I thought this section would be a warning for all extended simulation designers to be as prepared as 
possible before staring the simulation, the fact is that is not entirely possible or even desirable. I found that because I 
had some flexibility during the simulation I was able to respond to the simulation participants’ (both students and 
actors) requests. However, the flip side to this lesson is that one must also be prepared for unforeseen contingencies 
and have a back-up plan. 
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Some of the most important observations were that the students were very motivated, very involved the 
simulation, and processed information at a deep level. I believe it was the level of fidelity of the project and the 
amount of control they were afforded that helped the learners to use their newfound knowledge of PT to develop 
higher levels of problem-solving and critical thinking. At the same time, the structure of a planned simulation was 
important in providing a safe learning space while still at a novice level of performance. Because these observations 
are anecdotal, I will conduct a study this spring to gather empirical data to investigate the types of learning 
outcomes- both cognitive and motivational that are promoted through the simulation experience. 
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Abstract 

This paper reports on the creation of meaningful learning environments on campus in which student 
teachers employ digital filmmaking techniques to amplify their learning, represent their understandings and 
communicate in new ways. Constructivist learning conditions were created whereby novice teachers learned in the 
same ways they must teach today’s media savvy children. Ill structured problems provided the context for designing 
diverse solutions using technology. The social construction of knowledge was emphasized as student teachers 
worked collaboratively to design and produce their ideas using film. Diversity was not only valued but became 
essential for completing filmmaking tasks and helping novice teachers to begin to break the “teaching the way we 
were taught” cycle. 

Student teachers: Once we completed the project we began to realize that the project wasn’t really about 
technology at all; technology was simply the means to an end. 

Introduction 

As we shift from an industrial to a knowledge-based economy, there are major implications for how 
children learn in schools and how new teachers must be prepared on campus. Beginning teachers enter classrooms 
populated by tech-savvy students who interact skillfully and fluently with technology and information in ways that 
are different from those of the teacher’s generation. These third generation students (Norton & Wiburg, 1998) use 
and interpret technology in very different ways. Student teachers need to experience the same type of inquiry-based, 
technology-rich learning experiences on campus that they will be expected to create for children in their classrooms 
(Jacobsen, Clifford & Friesen, 2002; Jacobsen & Goldman, 2001). Workshops on how to operate the computer fail 
to provide sufficient time or opportunity for teachers to reflect on their beliefs and practices and to explore how 
technology can be woven into a meaningful learning environment (Sprague, Kopfman & de Levante Dorsey, 1998). 
Student teachers need intentional mentoring and support in the use of technology-enhanced, inquiry-based learning 
methods on campus and in their field experiences in order to resist the urge to “teach the way they were taught” as 
they encounter the many daily stressors and ill structured problems that characterize teaching as they begin their 
professional careers. 

A key challenge for faculties of education is to graduate new teachers who can embrace pedagogical and 
technological change and respond fluidly when presented with multidimensional, ill-structured problems (Jonassen, 
1997). In this paper, we report on the creation of meaningful learning environments on campus in which student 
teachers employ digital filmmaking techniques to represent their understandings. Filmmaking experiences provide 
opportunities for student teachers to wrestle with authentic, meaningful and challenging instructional problems from 
which they can then learn how to design diverse instructional interventions for their own students. We believe that it 
is through lived experiences designing with technology that novice teachers can begin to adopt changed dispositions 
toward teaching and learning and begin to integrate innovative approaches to problem solving with technology in 
their work with children. 

Our assertion, that faculties of education must graduate teachers who can respond and be sensitive to the 
myriad of complex and diverse perspectives and emerging technologies that they will face as they invite 
increasingly tech-savvy students to learn with them in expanding and distributed learning environments, is well 
supported by research (Palloff & Pratt, 2001; Willis & Mehlinger, 1996). Our second assertion, based on research 
and experiences teaching with technology on campus (Jacobsen, et ah, 2002, Jacobsen & Lock, 2003) and our 
experiences supervising student teachers in the field, is that intentional mentoring and support does enable novice 
teachers to implement technology-enhanced inquiry projects for learning in spite of the many challenges inherent in 
this pedagogical approach. 
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An Inquiry-based Teacher Preparation Program 

Unlike well-structured textbook problems, the ill-structured, real-world problems of professional practice 
are not pre-defined but emergent (Jonassen, 1997). In 1996, the Faculty of Education at the University of Calgary 
dismantled its textbook and course-based approach to teacher preparation in favour of an inquiry-based, field 
oriented and student centered program. The majority of teacher candidates who are accepted into this Calgary, 
Alberta program have completed a three or four year bachelor’s degree. The Master of Teaching (MT) program at 
the University of Calgary is a two-year professional teaching program leading to a bachelor of education degree. 

The MT Program philosophy is based on thinking differently about teaching practice. The MT Program was 
conceptualized on the understanding that teaching is largely a non-rationalized practice that resists step-by-step 
methods and solutions. From this view, teaching certainly requires content knowledge and skills; however, it is not 
simply defined as the application of a predefined set of skills and discrete disciplinary knowledge to achieve 
curriculum objectives. Student teachers participate in case, field and professional seminars, in which they 
interrogate curriculum and contexts, learners and learning, teachers and teaching, and explore the ethical, social and 
political contexts that frame the teaching profession. 

Integrating Technology Across the Curriculum 

The context for inquiry in this case study is a second year, semester four special topics seminar, entitled 
“Integrating Technology Across the Curriculum.” The two of us team taught this seminar to thirty-five students. No 
longer do we cycle students through a course on computer applications, as is the perceived need and design in many 
conventional four-year bachelor of education programs. A technology survey revealed that we were working with a 
more experienced and technology savvy population of undergraduate students who already have experience using 
email, word processing and conducting research using the Internet (Jacobsen & Clark, 1999, 2000). Instead, learning 
with technology is incorporated into field, case and professional seminars and is offered as an elective during the 
final semester. In Winter 2002 and 2003 we immersed student teachers in a culture of inquiry focused on 
technology integration across the curriculum (Jacobsen, et ah, 2002). We created constructivist learning conditions 
whereby novice teachers learned in exactly the ways they must teach today’s media savvy children. The five 
interdependent attributes of meaningful learning, described by Jonassen, Peck, and Wilson (1999) as intentional, 
active, constructive, authentic and cooperative, characterized the learning environment we felt was critical for 
student teacher learning and professional awareness. 

The social construction of knowledge was emphasized; student teachers worked collaboratively in groups 
to complete focused technology tasks designed for school children, and then built upon this understanding to 
develop rich, instructional plans for teaching. A theoretical framework and rationale for required group work is 
provided by Vygotsky (1986), who argued "... the true direction of the development of thinking is not from the 
individual to the social, but from the social to the individual” (p. 36). Filmmaking requires students to draw upon a 
diverse range of talents and skills. “What the child can do in cooperation today he can do alone tomorrow. 

Therefore the only good kind of instruction is that which marches ahead of development and leads it” (Vygotsky, 
1986, p. 188). We deliberately required student teachers to work together on complex problems that are difficult for 
one person to solve to promote the idea of professional practice being an ongoing conversation that occurs in a 
community of inquiry. 

Filmmaking tasks were designed to be meaningful, challenging and authentic. Tasks were inclusive; no 
matter what the technological starting point, all students had a role and made a contribution to the final outcome. We 
structured the task to give both clear guidance and flexibility in order to honor student ownership in the execution of 
the final product. Students were cast as members of a film team whose current assignment was to create a new 3- 
minute video of a well known story (i.e., legend, myth, or fairy tale) that could be shared on the internet. Students 
assigned each other roles, made decisions, booked equipment and human resources, identified and solved problems, 
and sought out additional skills and expertise when needed. Assessment rubrics were designed such that students 
were evaluated using a range of multi-disciplinary outcomes. A premium was placed on the clarity and elegance of 
student products. Recognizing that the work is never quite finished is at the heart of a meaningful inquiry project. 
We observed high student engagement and investment in the filmmaking tasks. Student teachers cared deeply about 
the quality of their work because it would be celebrated publicly with their peers, not just privately with their 
teachers. 

In a reflection on the creation of “Icarasaurus”, a film retelling of the ancient myth about Icarus, McKie, 
Smith, Milner, and Green (2002) describe how their group had a variety of opportunities to create projects and learn 
about technology through constructivism. The four member team drew upon the group’s diverse strengths, from 
acting to technology fluency to set design to planning, in order to achieve their goal of a three minute movie 
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published online. They researched and compared two published versions of the Icarus myth, created storyboards of 
these existing stories, wrote their unique version of the myth, created a production storyboard and shot lists, 
designed an elaborate set and cast a main character (i.e., plastic dinosaur), and began shooting raw video. The group 
published photo evidence of their filming and editing process online with the final video version of their story. The 
following is an excerpt from an online reflection about the learning process (McKie, Smith, Milner, and Green, 
2002). 

This was an invaluable experience because we were able to design units and activities that we 
could use in our own classrooms (in the near future), as well as experiencing what it is like for 
students to create and learn with the infusion of technology. We were expecting to learn a lot 
about technology and the process of integrating technology in meaningful ways. What we didn’t 
expect was to have our outlook on teaching and learning change based on what we discovered 
when technology and constructivism collide. 

The first revelation that we had was how much fun this project was. The guidelines put forth in the 
storytelling (filmmaking) focused task allowed for a lot of choice while working within a set 
framework that had clearly defined expectations. Choice and freedom within a clear set of 
guidelines allowed for our creativity to bloom as we were not concerned with unknown 
expectations. 

The entire process was one of creation (from choosing literature to designing sets for the movie) 
where we were allowed to take the project as far as we wanted to go, and of representation by 
using the digital media tools we thought would represent our story in the most powerful and 
meaningful ways. While we were learning about the technology and how to project our vision, 
each of us also saw the amazing potential for our own classroom with the high level of 
engagement and links to the curriculum. Filming Icarusaurus together was so much fun because 
we were exploring what it truly means to learn in a constructivist classroom and we wouldn’t have 
learned half as much without completing the focused task ourselves. 

The second realization was that group work could be an enlightening learning experience where 
we could learn from and teach one another. This project was created in a learning environment 
where we were expected to learn, plan and problem solve as a team, using skills and technology 
that were new to many of us. This allowed each of us in our diverse group to showcase our talents 
and expertise while feeling comfortable enough to take risks in learning and grow tremendously as 
both learners and future teachers. Project-based learning is often more open-ended and has the 
potential to appeal to all types of learners so that success and confidence follows in its wake. This 
was evident within our group where each of the members’ interests, background knowledge and 
learning styles were honored and explored throughout the planning, filming and editing stages. 

This helped us to see the potential for group-work in our own classrooms where students learn 
both from and with one another, and diversity is a necessary ingredient for group success. 

Finally, we truly began to understand how exciting technology could be when infused with 
meaningful and challenging learning. Once we completed the project we began to realize that the 
project wasn’t really about technology at all; technology was simply the means to an end. 

Utilizing technology in telling a story allowed us to be much more than writers and storytellers; it 
allowed us to be planners, organizers, set designers, puppeteers, camera people, artists, directors, 
editors and problem solvers- real life skills that will serve us well in future projects. We had to 
negotiate individual expectations, make complex decisions, use technology to communicate and 
exchange files, and meet often as a group. 

In closing, this project was an amazing first step on the journey that we took in the ICT Special 
Topics seminar that helped to guide our understanding of the role that technology should be 
playing in schools and learning today. By becoming students again and experiencing the power of 
constructivism through meaningful collaborative work with a multi-disciplinary focus, we began 
to see how learning could be enhanced by integrating technology, rather than by teaching 
technology for the sake of technology. 
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The “Icarasaurus” project (McKie, Smith, Milner, and Green, 2002) exemplifies the type of work we encouraged 
students to engage in and commit to during the seminar. Members of this group transferred the experience and 
energy created in this project to subsequent, more complex filmmaking projects that they completed with other peers 
during the seminar. We have had the good fortune to continue our work with two of these beginning teachers, and 
have observed how they carried the experiences of the focus tasks and integrated units of study into their actual 
practice with elementary children. We have observed how the beginning teacher’s awareness of instructional 
planning and differentiated instruction have matured as they confront the challenge of structuring filmmaking tasks 
that are inclusive of all students’ skills. The beginning teachers initially did not require the pre-planning and 
storyboarding, and just let the kids loose with the camera. Through experience and by assessing the students’ movie 
projects, these beginning teachers have learned with the children the importance of stressing process over product 

Jonassen’s (1997) model for solving ill-structured problems presents an approach to problem solving that 
provides an additional context for considering and understanding the intentional design of our focused tasks, 
including the storytelling and filmmaking task. His model includes various processes as follows: (1) Articulate 
problem space and contextual constraints; (2) Identify and clarify alternative opinions, positions, and perspectives of 
stakeholders; (3) Generate possible problem solutions; (4) Assess the viability of alternative solutions by 
constructing arguments and articulating beliefs; (5) Monitor the problem space and solution options; (6) Implement 
and monitor the solution; and (7) Adapt the solution (pp. 79-83). Jonassen’s first two steps correspond to problem 
formulation which is a prerequisite to problem solving. In the filmmaking task, we articulated the problem space and 
provided contextual constraints (i.e., you are part of a film team charged with the task of retelling an ancient myth in 
a three minute movie that is published on the web). The students were required to analyze the problem from the 
perspective of each group member, and generate possible solutions. Students created storyboards of existing myths 
and legends prior to developing and pitching their own idea for a retelling. Based on peer feedback, the film team 
created the final production storyboard. Hence, students assessed the viability of alternate solutions by constructing 
arguments based on experience. Managing the filmmaking project required pre-planning, organization and active 
problem and project management. Shooting raw video and editing film were both part of implementing plans, 
altering plans and monitoring the outcome. Peer review required each group to reflect upon the film they created, 
and to consider ways to refine their work. Knowledge and experiences with technology cannot be given to 
students. We actively engaged our students in our own thinking and reflection on technology integration as they 
were doing with technology. We interrupted each other’s presentations to make explicit to the students key points 
that were made, recognizing that the “obvious” about constructivism isn’t so obvious to the novice (or to many 
experienced teachers). We realized that we could only share stories from our experience, that we could not hand 
over our own experiences to our student teachers. Instead, we had to work together in an inquiry culture to build 
knowledge about technology, teaching and learning together. “Learning from teaching through inquiry assumes that 
beginning and experienced teachers need to engage in similar intellectual work. Working together in communities, 
both new and more experienced teachers pose problems, identify discrepancies between theories and practices, 
challenge common routines, draw on the work of others for generative frameworks, and attempt to make visible 
much of that which is taken for granted about teaching and learning” (Cochran-Smith & Lytle, 2001). Filmmaking 
enabled us to entice students to engage in a deeper level in learning about learning with us. Through projects like 
Icarasaurus, student teachers learned first hand through experience, and deepened this understanding of instructional 
practice through reflection and dialogue in a community of inquiry. 

Practical Experience also shows that direct teaching of concepts is impossible and fruitless. A teacher who 
tries to do this usually accomplishes nothing but empty verbalization, a parrotlike repetition of words by the child, 
simulating a knowledge of the corresponding concepts but actually covering up a vacuum (Vygotsky, 1986, p. 150). 

We resist the urge to lecture about constructivism and technology, choosing instead to interact 
meaningfully with our students. Our learning space was a studio rather than an instructor lead classroom modeling 
our understanding that the arts emphasize the creation of coherent structure, encourage multiple solutions to 
problems, prize innovation, rely on the use of judgment, and depend on the use of sensibility (Eisner, 1998). Our 
learning environment supported play, adventure and exploration with technology. We resisted the urge to lead whole 
class sessions on how to work the camera, or how to publish learning objects on the web. Instead, in response to 
needs that arose from students’ engagement in the projects, we did small group teaching. 

Students needed access to our thinking about constructivism, as well as first hand experiences in inquiry- 
based, technology enhanced learning environments, in order to create similar learning experiences for their own 
students. We reflected on the learning environment with them, helping them to make sense of their own learning 
experiences. We described our design so student teachers understood the type of instructional planning and 
dispositions that allowed this type of inquiry-based learning to happen. We facilitated professional conversations 
about how to understand and translate first hand experiences into personal models of informed practice. 
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When student teachers grapple with the task of producing a short film of their ideas they are developing 
multiple literacies that go well beyond text and graphics. Multiple literacies relate to the ability to construe meaning 
in dominant media forms used in a culture to create and convey meaning (Eisner, 1998). Because film is prevalent in 
our culture, learning how to read and create video is critical to informed citizenship in this century. Imagination is 
central to the process because we cannot know through language alone what we cannot imagine (Eisner, 1998). We 
designed the filmmaking tasks as ill-structured problems, or open-ended tasks, for which there are many possible 
“solutions” or responses. Ill- structured problems typically do not have a single “right” solution, but multiple possible 
solutions. Jonassen (1997) argues that “since ill-structured problems typically do not have a single, best solution, a 
learner’s representation of it should assume the form of an argument. . . The ‘best solution’ is the one that is most 
viable, that is, most defensible, the one for which the learner can provide the most cogent argument” (p. 81). 
Eilmmaking provide an opportunity for students to grapple with ill-structured, open ended problems, amplify their 
learning, represent their new understandings and communicate in ways that they have only been passive consumers. 
Students become critical consumers of the messages and media images created by others and develop as 
connoisseurs and active creators of their culture through the design and production of their own messages. 

The Compelling Nature Of Film 

The decision to offer our student teachers an opportunity to engage in filmmaking was intentional. We had 
experienced the excitement and value of engaging high school students in filmmaking using Apple Computer’ s 
iMovie software. We knew from extensive work with children and technology that authentic learning experiences 
that allow students to develop meaningful products that can be shared with an audience beyond the teacher (Cohen 
& Riel, 1989) not only capture students’ imaginations but can also change students’ lives. Eor example, our work 
with at risk youth helped us to understand the power of visual media to entice these students back into formal 
learning and empower them to tell their important stories in non-text ways. Designing and producing ideas using 
digital media liberates all students from conventional school structures that tend to still privilege reading, writing 
and mathematics. 

Erom our experiences with emancipated learners, we knew that filmmaking would draw on a broad skill set 
as it requires “. . . a collaborative process that brings together talent and energy of many artists from a variety of . . . 
backgrounds” (Theodosakis, 2001, p. 92) to complete a project. The need for collaboration becomes obvious to 
students as they take on filmmaking rather than having to be imposed by the teacher. Eilmmaking requires students 
to articulate individual talents and interests and then pool their diverse skills in order to make good choices as group 
members. Roles are not arbitrary. While one member might be able to envision the project and pitch it well, another 
member may have the graphic skills required for set design, etc. None of the tasks are hypothetical or contrived as 
all are essential for the completion of the final product. While one may initially think only of the roles of editor, set 
designer and cameraperson, Theodosaskis (2001) suggests five distinct roles in each of the five phases of film 
development. These roles should support a range of learning styles, assuming the task has been structured in such a 
way to encourage and value collaboration. 

Engaging students in learning activities such as filmmaking allows them to begin to understand 
the “. . . processes that experts use to handle complex tasks,” (Collins, Brown, and Newman, 1989, p. 457) and to 
join that community of expertise by attempting to solve ill-structured problems as the expert might. Lave and 
Wenger (1991) extend that sense of expert practice, suggesting that a novice’s “. . . ability to understand the master’s 
performance depends on engaging in the performance in congruent ways, suggesting “. . . if learning is about 
increased access to performance, then the way to maximize learning is to perform, not to talk about it” (p. 21). The 
distinct roles, identified by Theodosakis (2001), and supported by pre-filming activities, such as brainstorming and 
storyboarding, help students to understand the complex nature of professional filmmaking and to become more 
critical consumers of the medium. 

Eurther, it has been our experience that filmmaking is both an invitation to join the community of 
filmmakers and a guide for the engagement in authentic tasks that constitute the process of making movies. 
Eilmmaking has the potential to support the social construction of knowledge stressed in the literature (Vygotsky, 
1978; Bereiter & Scardamalia, 1993) because it requires the diverse skills suggested above and it encourages 
meaningful group work and authentic collaboration. 

We view film as “. . . a powerful tool for sharing, understanding, educating, documenting, indoctrinating, 
and entertaining” (Crichton, in Press) and we recognize that “Until recently, film has been a passive experience for 
the majority of us. We went to movies and watched them; few of us considered actually making them” (Crichton, in 
Press). We regard iMovie as an enabling piece of software with high usability. Students of all ages can quickly 
understand the conceptual design of iMovie and begin editing raw video within an hour or two. However, student 
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authoring using iMovie requires that educators recognize that multimedia projects such as film are as real and 
important demonstrations of understanding as text can be. 

Based on observations of children using Apple Computer’s iMovie software to create powerful 
representations of ideas important to them, we decided to make this experience available to future teachers. iMovie 
is a powerful desktop filmmaking tool that supports the development of multiple literacies, imagination, creativity 
and thinking. The usability of this ergonomically elegant videoediting tool enabled us to realize the benefits of 
filmmaking for student teachers. It is not that other digital filmmaking tools do not exist. They do. However, novice 
users can understand and use the iMovie interface and firewire data transfer quickly and easily, and then the tool 
fades into the background as the filmmaking process itself becomes central. 

Because of the demands iMovie places on the hardware, it forces an elegance of thought and design. Two 
to three minute movies become the standard as considerable amounts of hard drive memory are consumed for 
storage of the data and the editing features (adding sound, titles, transitions, etc.). This shift from quantity to edited 
quality, encourages the filmmaker distill his/her thoughts and to engage in meaningful inquiry as to what is 
important in the message, both forcing and encouraging wise editing. Eisner (1998) writes eloquently of the 
importance of the editing process, explaining “. . . there is nothing so slippery as a thought. . . . Working with a form 
of representation [text, video] provides the opportunity to stabilize what is ephemeral and fleeting. It gives students 
an opportunity to hold onto their thinking” (p. 27). Further he suggests “editing allows one to refine one’s thinking, 
to make it clearer, more powerful, and, not least, to appreciate the happy results of creativity. It allows one to confer 
a personal signature to a public product” (p. 27). We know that digital editing lets the filmmaker revise, edit, 
modify the images, allowing her/him to take an idea further and to incorporate other digital media to support the 
project. 

We have observed that filmmaking allows for the power and spirit of play to come forward. The original 
footage is available, so one can plan and experiment with images, both graphic and auditory, until the desired results 
are achieved. These images can be shared, collaboratively editing and modified, with the original filmmaker 
retaining access to the initial footage (Crichton, 2003). This allows one to manipulate the original content 
extensively, even to the point of throwing the work away as the original footage is safe on the tape in the camera that 
filmed it, allowing the filmmaker to start all over again with the editing process. 

However, we cannot emphasize strongly enough the importance of meaningful instructional tasks that 
focus on complex, ill-structured problems with multiple possible solutions. Jacobsen, et al (2002), have noted that 
typically “. . . task[s] involving technology tend to involve a fairly low level of thinking and research, focusing 
heavily on the presentation of final products rather than on thinking differently, rigorously and effectively at every 
stage . . .” (p. 2). Simply adopting iMovie as a neat new gadget in the classroom tool set will not in itself promote 
innovative learning. Instead, sound instructional tasks designed to incorporate the process of filmmaking and the 
type of ill-structured problems encountered by teachers can get beyond those low level skills by encouraging 
students to engage in the personal construction of meaningful learning and supporting the range of multiple 
intelligences and learning needs suggested by Gardner (2000). 

Filmmaking honors process over product, especially if the tasks that it supports are tied to sound learning 
outcomes. We note that filmmaking using iMovie supports the “just in time” approach to learning with technology 
that is consistent with our view of technology integration as it does not require extensive workshops or training. 
Rather, we have found that students learn the specific skills required to make their films as they engage in the 
project design itself. However, we did ensure that rich, just in time resources and support were available. We 
recognize that challenging students to work through problems is one thing, but that we cannot risk frustrating them 
with either faulty equipment or missing pieces of critical how-to information. Therefore, we knew that books, 
manuals, support documents, and human resources were available to the students. 

Discussion 

We believe that in order for more school children to experience learning as knowledge building in 
technology enabled environments, their teachers need to experience and understand learning in these ways. Student 
teachers and their experienced colleagues need learning experiences that model the cultural shift from being passive 
receivers of information to becoming active builders of knowledge. So, based on this reasoning, we offer these three 
reasons for why student teachers require the type of authentic and meaningful learning experiences that are provided 
by filmmaking. First, as future professionals, novice teachers need opportunities to analyze and solve ill-structured 
problems that draw upon their diversity, their creativity and their imagination. Teaching is not a step-by-step, well- 
structured problem domain, nor is it a solitary endeavor. Second, new teachers will be working with media savvy 
students who are creating, and can create, in this medium. Beginning teachers need to be media literate and author 
using multiple media in order to be a scholarly guide to their students. Third, and most important, new teachers can 
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provide leadership and champion innovative practices in the field. We have shared two stories about new teachers 
who transferred their filmmaking experiences into reformed practice for elementary students. In one case, a student 
teacher actually carried the computer and digital video camera into the grade one classroom to demonstrate with the 
children how filmmaking was possible in the very early grades. The partner teacher, who was initially uncertain 
about whether filmmaking would be accessible to the students, was able to learn from her protege. In many cases, 
student teachers cannot simply model and mimic their partner teacher’s practices with technology because many 
experienced teachers are grappling with meaningful integration themselves. Instead, student teachers who have 
experienced meaningful technology design and development, like those who completed filmmaking and other 
focused tasks in our seminar, bring value and innovative practice to the table and can make an immediate 
contribution to changed practice by championing innovation in the classroom. 

Filmmaking enabled our students to discover for themselves the importance of diversity - in terms of 
learners’ tasks, skills, and experiences. And this understanding is essential for breaking the “teaching the way we 
were taught” cycle. In our ongoing work with our second year teachers we have had the privilege to observe how 
they have made filmmaking their own. They can clearly articulate the importance and power of engaging their 
students the types of tasks we value. They place premium on creativity and imagination, linking both to authentic 
demonstrations of knowledge and questioning professionally what is worth knowing and understanding. They have 
demonstrated to us, and more importantly to their students and colleagues, that movie making is an exciting entry 
point for technology integration, recognizing that “The starting place is where ... [a person is] at the moment ...” 
(Jacobsen, et al, 2002, p. 11) and that the diversity of the starting point is "... not a problem to overcome but a 
strength to draw on ...” (p. 11). 

Based on the outcomes of our work with student teachers, and the transfer we have seen to first and second 
year teaching practice, we believe we have developed a sustainable model for educational reform. One teacher at a 
time, one classroom at a time, we can make a difference in the rich learning opportunities that are available to 
children. 



It goes on one at a time, 
it starts when you care 
to act, it starts when you do 
it again after they said no, 
it starts when you say We 
and know who you mean, and each 
day you mean one more. 

Marge Piercy, “Low Road”, cited in Palmer (1998) 

We live our work with beginning teachers, one at a time, in authentic mentoring relationships that are 
cultivated in a culture of inquiry. We see our work as intentionally transforming the learning opportunities available 
to today’s children. We continue to work with graduates who are now teaching and conducting research on 
integrating technology across the curriculum. We are working with a second year elementary teacher on classroom- 
based research projects that engage elementary school children in multiple forms of literacy by exploring authoring 
and media. We have worked with a student teacher who made filmmaking the focus for her final semester inquiry 
project and felt empowered enough to champion its use in a school that had not used it before. She read 
Theodasakis’ (2001) book. The Director in the Classroom, and borrowed filmmaking equipment to take to the 
school. She studied the impact of filmmaking on primary students’ learning and engagement. As we move forward 
with this approach to technology integration in teacher preparation, we see each interaction and each project as a 
possible ripple that will extend outward. Who knows how far it will go. 

Future Research 

New teachers enter the profession filled with hope and expectation. However, too many new teachers find 
themselves isolated in classrooms surrounded by colleagues who may not have the necessary time, skills or 
dispositions to support or encourage new colleagues for continued growth or innovative technology practices. The 
induction of new teachers into the profession should not be left to chance. While student teachers may experience 
inquiry-based, technology-enhanced learning on campus, they need ongoing support as new teachers to implement 
these approaches in the classroom. The next stage of our research is to investigate the nature of onsite and online 
mentoring and support which enables first and second year teachers to implement inquiry-based, technology rich 
learning experiences with today's students. 



199 




References 

Bereiter, C. & Scardamalia, M. (1993). Surpassing Ourselves: An Inquiry into the Nature and 
Implications of Expertise. Chicago: Open Court. 

Cochran-Smith, M. & Lytle, S. L. (2001). Beyond Certainty: Taking an Inquiry Stance on Practice. In 
Lieberman, A. & Miller, L. (Eds.). Teachers Caught in the Action. New York: Teachers College Press. 

Collins, A., Brown, J., & Newman, S. (1989). Cognitive Apprenticeship: Teaching the Crafts of Reading, 
Writing, and Mathematics. In L.B. Resnick (Ed.), Knowledge, Learning and Instruction, (pp. 453-494). Hilldale, 
New Jersey: L. Erlbaum Associates. 

Crichton, S. (in press). If 1 Were Eight Again ... The Power of iMovie. AGATE - Journal of Alberta 
Gifted Education. 

Crichton, S. (2002). Review Essay: The Director in the Classroom: How Eilmmaking Inspires Learning. 
International Electronic Journal for Leadership in Learning, 6(16). Retrieved July 14, 2003, from 
http://www.ucalgary.ca/~iejll/volume6/crichton.html 

Eisner. E. (1998). The Kind of Schools We Need: Personal Essays. Portsmouth, NH: Heinmann. 

Jacobsen, D. M., & Lock, J. V. (in press. Sept 2003). Technology And Teacher Education Eor A 
Knowledge Era: Mentoring Eor Student Eutures, Not Our Past. Journal of Technology and Teacher Education, an 
AACE Journal (http://www.aace.org/). 

Jacobsen, D. M., Clifford, P., & Eriesen, S. (2002). Preparing Teachers for Technology Integration: Creating a 
Culture of Inquiry in the Context of Use. Contemporary Issues in Technology and Teacher Education [Online 
serial], 2(3). Available: http://www.citejoumal.org/vol2/iss3/currentpractice/article2.cfm 

Jacobsen, D. M., & Goldman, R. (2001). “The Hand-Made’s Tail: A Novel Approach to Educational 
Technology”. In Barrell, B. (Ed.). Technology, Teaching and Learning: Issues in the Integration of Technology, p. 
83-112. Calgary, AB: Detselig Enterprises Ltd. 

Jacobsen, D. M., and Clark, W. B. (2000). New Meets New Year Two: Integrating Technology into Inquiry- 
Based Teacher Education. Proceedings of SITE 2000: Society for Information Technology and Teacher Education 
10th International Conference, San Diego, CA, Eebruary 8 - 12. 

Jacobsen, D. M., & Clark, W. B. (1999). New Meets New: Fitting Technology To An Inquiry-Based 
Teacher Education Program. Proceedings of SITE 99: Society for Information Technology and Teacher Education, 
10th International Conference, San Antonio, Texas, Eebraary 28 - March 4. 

Jonassen, D., Peck, K., & Wilson, B. (1999). Learning With Technology: A Constructivist Perspective. 
Upper Saddle River, NJ: Prentice Hall. 

Jonassen, D.H. (1997). Instructional Design Models for Well-Structured and Ill-Structured Problem- 
Solving Learning Outcomes. Educational Technology: Research and Development, 45(1), 65-95. 

Lave, J. & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. New York: 
Cambridge University Press. 

McKie, K., Smith, K., Milner, A., and Green, T. (2002). Icarusaurus: Retelling Stories Via Video. 

EGallery: Exemplary Student Scholarship, 5(1). Retrieved October 15, 2003 from 
http://www.acs.ucalgary.ca/~egallery/volume5/icarusaurus/task.html 

Norton, P., & Wiburg, K. (1998). Teaching with Technology. Orlando, Elorida: Harcourt Brace College 
Publishers. 

Palloff, R. & Pratt, K. (2001). Lessons from the Cyberspace Classroom. San Erancisco, CA: Jossey-Bass. 

Palmer, P. (1998). The Courage to Teach: Exploring the Inner Landscape of a Teacher's Life. San 
Eransico, CA: Jossey-Bass. 

Sprague, D., Kopfman, K. & de Levante Dorsey, S. (1998). Eaculty development in the integration of 
technology in teacher education courses. Journal of Computing in Teaching Education, 14 (2), 24-28. 

Theodosakis, N. (2001). The Director in the Classroom: How Filmmaking Inspires Learning. San Diego, 
CA: Tech4Learning, Inc. 

Vygotsky, L. (19^6). Thought and Language. Cambridge, MA: MIT Press. 

Willis, J. W., and Mehlinger, H. D. (1996). Information technology and teacher education. In Sikula, 
Buttery & Guyton (Eds.), Handbook of Research on Teacher Education (2"“* Edition, pp 978-1029). New York: 
Macmillan Publishing. 



200 




The Effects of Response Constraints and Message Labels on Interaction 
Patterns and Argumentation in Online Discussions 
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Sunyoung Joung 
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Abstract 

This study tested the effects of constraint-based argumentation in threaded discussions. In one group, 
students posted messages using prescribed response categories. In another group, students pre-classified messages 
by inserting response labels into message headings, enabling students’ to examine the structure of their arguments. 
The control group conducted debates with no knowledge of the response categories and no message labeling. Using 
event sequence analysis to identify interaction patterns in the threaded discussions of each group, this study found 
that message labeling reduced the proportion of arguments challenged by the opposing team and as a result, 
reduced the proportion of arguments elaborated with explanations. Message labeling may have heightened the 
perception of critical responses as excessively confrontational among the mostly female participants. 

Introduction 

Numerous studies have demonstrated that online learning environments provide better opportunities for 
more learner-to-leamer interaction and facilitate more reflective and critical thinking within learning communities 
(Ravits, 1997; Collins & Collins, 1996; Ward & Tissen, 1997). Argumentation is recommended as one of the 
collaborative inquiry-based learning strategies to increase student critical thinking skills in online settings (Derry et 
ak, 2000). These skills involve the processes of building arguments to support a position, and considering and 
weighing evidence and counter-evidence in developing supporting arguments. 

Various techniques have been developed to scaffold argumentation in online discussions. Derry et al. (2000) 
suggested specific tools that reduce the difficulty of teasing out argumentative structures in group interactions to 
support better understanding of task-related mental structures and arguments. The ability to visualize the complex 
discourse structures can ultimately support problem solving and learning communications (Turoff & Hiltz, 1998). 
Such tools can perform some part of the tasks for students but also assist students’ abilities by changing the nature or 
difficulty of the task (Cho & Jonassen, 2002). Among the tools that have been found to support argumentation, 
structural scaffolding is provided by constraining the types of messages that can be posted and requiring students to 
pre-classify or label messages by function using prescribed response labels (Duffy et ak, 1998; Cho & Jonassen, 
2002). For example, the Belvedere system (Suthers, 1998; Suthers et al, 2002) presents a constraint-based 
argumentation scaffold consisting of predefined argumentation constraints (hypothesis, data, principles, and 
unspecified) and links (for, against, and). 

To improve our understanding of constraint-based computer- supported collaborative argumentation (CSCA), 
these tools must be examined under controlled experiments. Current CSCA research has focused primarily on 
graphical tools that support collaborative production of concept/knowledge maps (Suthers, 1998; Cho & Jonassen, 
2002, Veerman, Andriessen & Kanselaar, 1999; Buckingham-Shum, MacLean, Bellotti & Hammond, 1997; Suthers 
& Hundhausen, 2001). These studies compare the effects of graphical tools to other communication tools such as 
chats and asynchronous bulletin boards used without argumentative scaffolds. At this time, no studies have 
compared the effects of threaded discussion tools with versus without constraint-based argumentation. Previous 
studies on threaded discussions with constraint-based argumentation and labeled messages are limited to findings 
from evaluation reports (Duffy, Dueber & Hawley, 1999; Sloffer, Dueber & Duffy, 1999) and not from controlled 
experiments. 

The purpose of this study was to examine how constraint-based argumentation with labeled messages affect 
students’ interaction patterns and argumentation processes in online threaded discussions. Using a controlled 
experimental design to examine the effects of response constraints and labeled messages, the objectives of this study 
were to compare the following outcomes: 

1) relative frequency of arguments, supporting evidence, challenges to arguments, and explanations for 
clarifying or elaborating stated arguments. 

2) overall level of interaction and the response rate to messages. 
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3) interaction patterns and percentage of arguments and critiques that elicit responses with supporting 

evidence, critiques from the opposition, and explanations. 

Method 

Participants. In this study, forty-three pre-service teachers participated in a series of five weekly debates on five 
educational issues. The students were largely undergraduate students enrolled in a course on educational technology. 

Treatment Groups. Students were randomly assigned to three treatment groups consisting of 15 students per group. 
Within each group were two male and thirteen female students. In control group 1, students received no instructions 
or constraints on the types of messages they could post to the debates. Students in group 2 were presented with a 
prescribed set of response categories - arguments, evidence, explanation, critique, and other process comments - 
used as a guideline on how to contribute to the debates (see Figure 1). In group 3, students received the same 
prescribed response categories, but were also required to insert corresponding labels into the subject headings (see 
Figure 2). The labels enabled students to see the structure of their arguments when viewing the discussion threads. 

Students were required to post a minimum of four messages per week. Within each group, students were 
assigned to a team to either support or oppose a position. In each debate, students debated issues that explored the 
appropriate use of educational technology. For example, students debated the position “Computers can substitute the 
role of the instructor for kids in the future”. 

Data Analysis. Previous studies that examine interaction patterns in online discussions have used content analysis 
(Gunawardena et ak, 1997; Newman, 1995; Levin, Kim & Riel, 1990) to gather simple event frequencies for 
comparing treatment groups. This study applied a more advanced approach to examining interactions by using event 
sequence analysis (Bakeman & Gottman, 1997) to identify patterns in message-response exchanges. Using event 
sequence analysis, this study identified prevalent patterns (see Figures 3) in message-response sequences by: 1) 
counting the frequency of specific responses to each type of message; 2) converting the response frequencies into 
transitional probabilities for each observed message-response interaction, e.g. probability of an argument eliciting a 
critical response versus a response with supporting evidence; 3) testing the statistical significance and strength of 
observed transitional probabilities to determine if they were significantly higher or lower than expected 
probabilities; and 4) translating the transitional probabilities into transitional state diagrams (see Figure 4) to provide 
a visual birds-eye view of interaction patterns. The analysis also included relative frequencies and response rates for 
each response category. The analysis and state diagrams were computed and generated with the Discussion Analysis 
Tool (Jeong, 2003). 

The discussion transcripts in Groups 1 and 2 were classified by two coders. Cohen’s kappa was calculated 
to establish inter-rater reliability of the coding. The reliability of the codings were very good for group 1 (k = .82) 
and group 2 (k=.87). The assessment was determined based on consensus between coders. Decisions about messages 
with different codes were reconciled in discussions between the coders. In group 3, messages were labeled by the 
students themselves. As a result, one coder classified the messages and a comparison of the coder’s classifications 
with the students’ classifications also produced a very good inter-reliability of k = .804. 

Theoretical Assumptions. The justification for examining the transitional probabilities between messages and 
responses is based on the assumptions of dialogic theory (Bakhtin, 1981). The theory posits that all possible 
meanings of a word interact, possibly conflict, and affect future meanings. Meaning is produced by the relationship 
between one utterance and another, and not by the utterance alone. Meaning is therefore affected, re-negotiated and 
reconstructed as a result of conflict arising from social interaction and “dialogic reasoning” (Sappo & Mononen- 
Aaltonen, 1998). As a result, the construction of knowledge depends on specific processes and social interaction 
patterns such as argument-critique, critique-explanation and argument-evidence. 

Results 

Relative Frequencies. The number of postings in Groups 1, 2 and 3 were 207, 300 and 265 respectively. 
These findings suggest that the use of constraints or prescribing response categories to scaffold argumentation may 
have helped to generate more postings. In addition, the results in Figure 5 suggest that message labeling in Group 3 
helped to produce proportionately more arguments and evidence than Groups 2 and 3. Group 3 posted 
proportionately more arguments (.52) than Group 1 (.27) and Group 2 (.35). Group 3 posted proportionately more 
evidence (.23) than Group 1 (.05) and Group 2 (.05). However, Group 3 posted fewer critical responses to arguments 
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(n = 30), and fewer explanations to elaborate on arguments (n = 27). Group 1 posted 70 critiques and 70 
explanations, and Group 2 posted 97 critiques and 79 explanations. 

Response Rates. The overall response rates to posted messages for Groups 1, 2 and 3 was .63, .60 and .56. 
This finding suggests that the use of prescribed response categories can reduce interaction between students given 
the response rates in the control Group was greater than the response rates in Group 2. The addition of message 
labeling in Group 3 may have further reduced the level of interaction. As a result, the previous finding that message 
labeling and constraint-based argumentation may produce more arguments and evidence must be tempered by its 
negative effects on the level of interaction needed to engage students in argumentative exchanges. 

Interaction Patterns. Comparing the state diagrams in Figure 4 shows that Group 3 engaged in substantially 
fewer ARG^CRIT exchanges than Groups 1 and 2. No apparent differences were found between Groups 1 and 2 
because students in Group 2 were not given strict instructions to restrict messages to the prescribed response 
categories. Of the responses to arguments in Group 3, only .12 of responses challenged the basis of the arguments 
compared to .45 in Group 1 and .44 in Group 2. The diagrams also show that a substantially smaller proportion of 
responses to arguments were elaborated with explanations (ARG^EXPL) in Group 3 (.07) compared to Group 1 
(.42) and Group 2 (.42). The low number of explanations can be attributed to the low number of critical responses to 
arguments given the assumption that only when something appears to be wrong that the need to explain and 
understand is felt and acted upon (Baker, 1999). In other words, the ARG^CRIT interactions serve as events that 
trigger explanations which in turn can lead students to a better understanding of arguments. 

The low rate of ARG^CRIT interactions in Group 3 may have resulted from the use of particular labels or 
sequence of labels (e.g. ARGo^CRITs, ARGs->ARGo) that heightened the perception of particular responses as 
personally and excessively critical and confrontational. Students may have avoided posting critical responses so as 
not to be perceived by other students as aggressive or dominating. This particular response to labels may be largely 
attributed to the nature of the group composition in which the majority of students were female. Studies have found 
males to be more likely than women to engage in argumentation, confront differences in opinions, and defend 
viewpoints (Tisdell, 1993; Vanfossen, 1998). Women tend to engage in more relational communication patterns that 
are supportive instead of confrontational (Tannen, 1990). This is consistent with the findings in Group 3 where the 
postings were mostly arguments and evidence posted in support of team positions. In another study that used 
message labeling (Jeong, 2003b), females were less likely than men to post critical responses, but only when 
responding to messages from females and not males. Interaction between genders helped generate more 
ARG^CRIT and ARG^EXPL interactions. 



Discussion 

Previous studies have shown that constraint-based CSCA can improve argumentation when applied with 
concept mapping tools. This study however found that constraint-based argumentation with labeled messages in 
threaded discussions can inhibit argumentation. Although the results in this study show that labeled messages 
produced more arguments and evidence, labeling reduced overall interaction between students and reduced the 
number of critical responses to arguments and evidence. As a result, this study found that labeled messages in 
threaded discussions can inhibit interaction and critical analysis of arguments. 

Although these findings can be largely attributed to the female response to labels, the findings nevertheless 
has important implications. Many concept mapping tools support argumentation without labeling the nodes and 
identifying its authors. As a result, constraint-based discussion boards could be implemented in the same manner by 
requiring students to pre-classify messages without displaying labels and author’s identity. Despite the findings in 
this study, labeled messages has the potential to help students examine the structure of arguments in discussions 
threads and quickly locate points of disagreement in order to foster more in-depth discussions. Message labeling also 
has the potential to be a practical method for collecting ever larger corpus of data for future research on 
collaborative models and processes. The problems identified in this study must be addressed to allow further 
exploration of the potential advantages of labeling as an instructional and research methodology. 

Some solutions to the problems identified in this study, when implemented and manipulated in controlled 
experiments, suggest directions for future research. Some of these solutions include the following: implement 
anonymous discussions, include labels that are facilitative and gender neutral, ensure more gender-balanced 
discussion groups, implement additional constraints on message-response sequences to ensure more argumentative 
exchanges, and assess performance to set appropriate expectations and encourage more argumentative exchanges 
between opposing viewpoints. 
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Figure 1 Response categories used to scaffold and label messages for argumentation 



ARG 


Argument in support of the proposition. For example, "Computers can substitute the 
role of the instructor when kids use computer-based tutorials". The subject heading 
for this argument might be something like "ARGs computers as tutors" 


EVID 


Evidence, specific examples, research studies, personal observations & experiences, 
or proofs. For example "I found an article that shows that..." or "In one course, I used 
a computer tutorial and I found that....". The subject heading for these messages 
might be "EVIDs study finds tutorials to be effective". 


GRIT 


Challenge or critique given arguments, evidence or any other responses from the 
opposing team. For example, "The personal experiences you've had with computer- 
based tutorials cannot be generalized to other students because ... " or "The study 
you cited was flawed because..." 


EXPL 


Explanation, elaborations or clarifications. For example "Computer-based tutorials 
perform the specific role of assessing students understanding just as an instructor 
can administer pop quizzes in a classroom" . 


OTH 


Other process comments or questions or requests for help, such as "What is a 
computer-based tutorial?" or "Thanks for responding to my message" or "I think we 
need to provide more evidence, such as jorrmal articles, textbook readings, personal 
observation & experiences from past courses, to show that computer-based tutorials 
perform just as well as an instructor?" or "What exactly did you mean when you said 
that....?" or "Can you give me more evidence?" 



Students were also required to insert tags (‘o’ = opposing, ‘s’ = supporting) at the end of each label to identify 
postings from the opposing versus the supporting team. For example, ARGs for an argument from the supporting 
team and CRITo for a critical response from the opposing team. 
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Figure 2 Illustration of threaded discussion forum for Group 3 with prescribed response categories and labeled 
messages 
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Figure 3 Event sequence matrices containing observed response frequencies, transitional probabilities and Z-scores 
values for Group 3 using constraint-based argumentation with message labeling 





iD 

CL 

< 


EVID 


CRIT 


EXPL 


OTH 


Replies 


No replies 


Givens 


%Replies 


%Givens 


Reply rate 


ARG 


56 


4 


10 


6 


6 


82 


59 


137 


.57 


.52 


.57 


EVID 


4 


8 


4 


1 


3 


20 


13 


32 


.14 


.12 


.59 


CRIT 


2 


0 


7 


3 


1 


13 


13 


30 


.09 


.11 


.57 


EXPL 


6 


2 


3 


6 


1 


18 


9 


27 


.12 


.10 


.67 


OTH 


1 


0 


3 


0 


8 


12 


23 


39 


.08 


.15 


.41 




69 


14 


27 


16 


19 


145 


117 


265 


__ 


__ 


.56 





ARG 


EVID 


CRIT 


EXPL 


OTH 


ARG 

EVID 

CRIT 

EXPL 

OTH 


.68 .05 .12 .07 .07 

.20 .40 .20 .05 .15 

.15 .00 .54 .23 .08 

.33 .11 .17 .33 .06 

.08 .00 .25 .00 .67 








ARG 


EVID 


CRIT 


EXPL 


OTH 


ARG 

EVID 

CRIT 

EXPL 

OTH 


5.70 - 2.22 - 2.27 -1 63 - 2.36 

- 2.66 4.95 0.17 -0.93 0.27 

- 2.44 -1.24 3.42 1.45 -0.61 

-1 .29 0.22 -0.23 3.23 -1 .01 

- 2.84 -1.18 0.59 -1.27 5.74 



Z-scores < -1.95 and > 1.95 identify transitional probabilities that were significantly below and above, 
respectively, the expected probabilities based on random chance alone at alpha = .05. 
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Figure 4 The transitional state diagrams for all three groups, summarizing the observed frequency of responses, 
transitional probabilities between message- response interactions, and z-score tests for probabilities significantly 
higher and lower than expected probabilities 

Group 1 - Control group Group 2 - Constrained by Response Categories 




Group 3 - Constrained by response categories with message labeling 




Each node in the diagrams represents a specific response/message category. The lines connecting the nodes illustrate 
the strength of the transitional probabilities between message and responses. The probabilities displayed in green 
and red designate probabilities that were significantly higher and lower, respectively, than the expected probability 
based on z-score values (alpha = .05). Within each node are the observed frequencies of the given message followed 
by the total number of observed responses to the given message. 
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Figure 5 The proportion of messages and responses for each response category by group 
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Instructional Design in the Real World 

Many instructional design (ID) tools and methods evolved during the 1940s, 50s, and 60s (Reiser, 2001). In 
the 1970s, these methods were solidified in several classic instructional design models. In the 1980s, instructional 
design continued to flourish, fueled by the introduction of personal computers and advances in cognitive 
psychology. However, during the 1990s, the field was challenged by new developments (Gustafson & Branch, 1997; 
Reiser, 2001). The performance technology movement pushed corporate training departments to consider other 
kinds of interventions instead of relying on instruction to solve performance problems. Proponents of the 
constructivist movement in education harshly criticized the use of behavioral objectives and other strategies that 
served as foundations for the field. To rein in e-learning development expenses, designers began to create prototypes 
before committing to costly full-scale production. Despite these challenges, the need for instructional design and 
development skills increased as “just-in-time” electronic performance support systems, distance education, and 
instructional applications of the Internet all exploded. 

However, researchers have discovered that practicing instructional designers seldom perform all the steps 
prescribed by classic instructional design models (Rowland, 1992; Wedman & Tessmer, 1993; Wiener & Vazquez- 
Abad, 1995). In response to these findings, several authors have called for the development of a “practitioner’s 
model” that reflects current practices (Tessmer & Wedman, 1990; 1995). 

In this article, we present instructional design as we practice it and as we see our peers practice it. This 
“practitioner’s model” has evolved from the literature on instructional design and an extensive examination of 
design artifacts from more than 20 years of combined experience in academic and corporate settings. Design 
artifacts ranged from macro-level documents such as project proposals to micro-level documents such as meeting 
agendas and storyboards. Through the process of continually looking at the research of others (for example, 
Rowland,1992; 1993; & Wedman, 1990; 1993; 1995), examining documentation of design decisions and design 
artifacts, and active conversing and reflecting on the processes we saw represented there, we have outlined a model 
of design that we feel accurately represents the way instructional design is practiced in the “real world”. 

Unlike the crisply delineated, step-by-step instructions found in cookbooks, we know that instructional 
design resists a tidy, easily replicated solution. Each instructional design project is, by definition, creating new work. 
If the work is not entirely new, perhaps the content is new, or there’s a new group of learners, or the delivery system 
has changed, or perhaps several things have changed. If nothing has changed, there wouldn’t be a need to do 
anything to the material. So instructional design isn’t similar to making banana bread - you can’t follow the same 
instructions in the same way each time and get equally great results. 

We agree that design involves problem solving (Rowland, 1993). As the designer works on a project, he or 
she collects information and makes tentative hypotheses about solutions. Expert designers often use a “template” to 
provide a mental framework for collecting this information and then fill in the “slots” with information (Rowland, 
1992). Eventually, the designer reviews which slots are filled and which still need to be figured out. At some point 
the designer will find there’s nothing more to add, either because experts and resources have yielded all the 
information that’s needed, or because of time or budget constraints. Throughout the process, the designer makes 
tentative decisions, develops design documents and other products, presents them to the client for feedback, and then 
revises them based on that feedback. 

We divide this knowledge-building cycle into five phases of instructional design; Define, Design, 
Demonstrate, Develop, and Deliver. Running through the five phase are five essential elements that form the 
building blocks to systematically designed instruction: learner needs and characteristics, goals and objectives, 
instructional activities, assessments, and evaluation. Instructional designers must create instruction with others as 
well as /or others. Various team members come from different professional cultures, each with their own set of 
expectations, vocabulary, timelines, rewards, and consequences. Designers need more than a step-by-step model to 
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prepare them for the collaboration required to successfully solve instructional problems in the team environment in 
which most designers work. 

Critical questions: The essential elements of instructional design 

Rowland (1992) suggests that instructional designers focus on asking good questions, rather than simply 
following the "steps" prescribed by traditional instructional design models. Throughout the instructional design 
process, instructional designers need to consider factors that impact the success of the instruction. These factors, or 
elements, are the component parts of instructional design and can be thought of as a set of questions. The answers to 
those questions inform the results of the design process. The design process starts with these considerations: 

1. What are the learners’ information needs, prior experiences, beliefs, and values relative to the topic of 
instruction? Instructional decisions should be made based on what we know about the learners ’ 
characteristics and informational needs. 

2. What changes in thinking or performance do we want to occur? How will we know if these changes have 
occurred? How will we provide opportunities for learners to engage in activities that may stimulate changes 
in thinking or performance? The designer examines these questions, determining the goals, assessments, 
and activities of the instructional intervention, recognizing that to be effective, instruction requires a match 
among the goals, assessments, and activities. 

3. How do learners respond to the materials? Are they effective? Good instruction is instruction that has been 
tested with a group of learners prior to distribution, and modified based on the results of a pilot test or 
formative evaluation. 

In examining these issues, we define the learners, their needs, the instructional objectives, the assessment, 
the instructional activities, and the evaluation. As instructional designers, our task is to ensure that: 

• Instruction is designed to meet the needs and characteristics of the learners. 

• The goals, activities, and assessments of instruction are in alignment. 

• The instruction has undergone /ormafi've evaluation prior to distribution. 

The relationship among these key ideas is illustrated in Figure 1. 

Figure 1 . The Essential Triangle of Instructional Design 




These five essential elements are addressed in almost all instructional design models (Gustafson & Branch, 
1997). Where traditional instructional design models include discrete stages for defining each of these element (for 
an example of one popular model, see Figure 2), we find that most projects unfold in a spiral rather than a linear 
fashion. Most traditional instructional design models require the designer to determine needs before writing 
objectives, write objectives before assessments, and assessments before the instructional strategy. That’s a fine 
strategy but we find that, in practice, designers actually might start by seeking answers to any or all of these 
questions. 
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Figure 2. The Dick and Carey model 
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Knowledge-building cycles: phases of instructional design 

As the we cycles through the stages of Define, Design, Demonstrate, Develop, and Deliver, we revisit the 
elements of the essential triangle again and again. Instead of addressing each of these elements in a set sequence, 
each of the elements is considered in each phase, building on the knowledge gained in previous phases. Throughout 
the design process, the designer reconsiders the elements, revising and refining them as he or she develops a deeper 
understanding of the audience, the outcomes, and the activities. As the spiral tracks the project through each of the 
phases, we move from a vague vision to a concrete, effective product. 

Figure 3. The phases of instructional design 




Fig K. Tht dfidogniMt prtceif. 
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In the Define phase, we begin to determine the learners, the needs, the overall goal, potential benchmarks 
of success (assessments), the product, and begin to think about how to determine if the program is effective 
(evaluation). This stage results in a Project Proposal. 

The Design phase involves the major planning effort to provide crisper audience definitions, confirmation 
of instructional needs, goals converted to high-level objectives, benchmarks converted to assessment plan, general 
activities converted to content and instructional strategies, and a plan for prototype testing and formative evaluation. 
This stage results in Instructional Design Documents. 
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In the Demonstrate phase, we continue to develop design specifications and to ensure quality as the 
preliminary media production begins. This includes determining that the materials, in development, are appropriate 
for the learners-, monitoring development of materials to ensure they meet audience needs-, making sure activities 
meet objectives-, ensuring assessments, included as planned, align with the materials, which match the instructional 
strategy as designed; and testing the prototype with a small group of learners to get feedback prior to large-scale 
development. This stage results in detailed Production Documents such as storyboards and templates and a 
prototype consisting of a “slice” of the instructional materials. This is a logical time to begin to evaluate the 
instruction. The evaluators consider the degree to which the materials met the audience need, whether materials 
were appropriate for the learners, whether objectives were met, whether assessments were clear and appropriate for 
materials, and whether materials were implemented as planned and run smoothly. 

As we work with the production team during the Develop phase, we continue to serve as an advocate for 
the learner. This includes reviewing the feedback from the prototype testing and making any changes to the product 
as needed. This phase also requires that we ensure that materials are appropriate for the audience characteristics and 
needs, activities meet objectives, assessments are included as planned, and materials match the strategy as designed. 
This phase results in a complete set of instructional material, which may undergo additional evaluation in field 
trials.. 

Finally, we reach the Deliver phase, presenting the materials to the client, possibly training the client team 
on the product, and making recommendations for future development. If there is be a summative evaluation, the 
client and evaluator needs to consider the degree to which the materials met the audience need, whether materials 
were appropriate for the learners, whether objectives were met, whether assessments were clear and appropriate for 
materials, and whether the materials were implemented as planned and run smoothly. This phase results in the 
successful conclusion of this instructional design project and, hopefully, the beginning of a long and fruitful 
relationship between the client and the design team. 

We’ve found that as we revisit the same elements, at different times, for different purposes, ideas generated 
in previous phases are built upon in iterative, knowledge building cycles (Rowland, 1992). 

“While [traditional] models may have iterative features that allow for a reconsideration of earlier design 
activity outputs, they emphasize closure of each component in the process to serve as input to the next component. A 
layered approach assumes that components of the ID process will be repeated to a greater degree of precision and 
sophistication in subsequent layers of the process. This repetition is not for the purpose of revision earlier 
components ... but of adding onto the work that was done earlier” (Tessmer & Wedman, 1990, p. 80). 

Decisions are not revisited in order to stall progress; on the contrary, decisions are revisited to allow us to 
move on. In this process, decisions are made, but often must be made quickly, before all of the relevant factors are 
known. Instructional designers must generate hypotheses and makes “best guess” decisions in order to move 
forward, then confirm or revise those decisions as their understanding grows. In terms of a spiral, in the initial phase, 
we’ve found that the designer’s understanding of the material is tiny, like the tight inner coils of the spiral. As the 
project moves through subsequent phases, the designer often revisits the same material but brings greater 
understanding each time, moving finally into outer layers of the spiral. 

Design as collaborative practice 

We talk about the cyclical nature of instructional design, spiraling through stages and seeking deeper and 
deeper understanding of key questions. But how does a designer move from one stage to another? How do we gain 
that deeper understanding of each of the key elements of design? Throughout the design process, we consider each 
of the essential elements of design. We spiral through the various phases of the design and development project, 
increasing our understanding as we spiral outward. The process through which we increase our understanding can 
be thought of as a collaborative “ASC cycle” (see Figure 3) because we: 

• Ask questions 

• Solve problems, and 

• Check for understanding. 
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Figure 4. The collaborative cycle 




Each phase begins with asking for certain information. The instructional designer has many, many 
conversations. In those conversations, the role of the instructional designer is to derive information that is 
applicable to answering the essential questions of design; Who are the learners? What are their needs and 
characteristics? What do they need to know? How can we tell if they know it? What activities can we provide to 
increase the probability of these learners gaining this knowledge? We assemble documents, observe processes and 
use other means to gather the information we need to create tentative solutions. We have conversations with a 
variety of people who can help answer these questions. 

We collect information and then retreat back to our “design shop”, desk, or laptop, and begin to make sense 
of the information we’ve gathered relative to the design process. Along with the instructional design team, we begin 
to generate tentative solutions. Under the best of circumstances, the ideas generated will be elevated to a level that 
could never have happened when working in isolation. Through “synergy”, the ideas produced by the group will be 
more than the sum of it’s parts. As we make sense of the information, we begin to create the deliverables of each 
stage in the design process: the project proposal at the Define phase, the design documents in the Design phase, the 
production documents and prototype in the Demo phase, the instructional materials in the Develop stage, and the 
complete instructional package in the Deliver phase. 

And at each stage, we check our tentative understanding through presenting the deliverables to others for 
input as to whether our interpretation, as manifest in the deliverables of each phase, is on target for the client, SMEs, 
and learners. We evaluate our ideas when we present a project proposal for consideration. We evaluate our “best 
guess” solutions when we present design documents for discussion, when we present storyboards, content outlines, 
and prototypes. We are confirming or modifying our design decisions and the products of these decisions, 
throughout the Define, Design, Demonstrate, Develop, and Deliver cycles. 

As we check our understanding as instructional designers, we gain additional information that will move us 
to the next phase in the instructional design process. We negotiate a contract in the Define stage that leads us to the 
Design phase. We gain approval of Design documents in the Design phase that allows us to begin creating more 
detailed representations of the instruction in the Demonstration phase. We gather input from clients, learners, and 
SMEs in the Demonstrate phase that allows us to move into the Develop phase. And as we gain approval of the 
materials developed, we Deliver the materials to our clients and learners. 

Instructional design projects often involve a client, a subject matter expert, and one or more individuals 
with specific media production skills, in addition to the instructional designer. The designer may need to interact 
with members of the target audience or other individuals who are familiar with the learners’ needs and 
characteristics. Collaborative interactions with these various groups serve three primary purposes: 

• to provide answers to the key questions of design 

• to allow solutions to emerge that would not emerge in isolation (through synergy) 

• to confirm or modify our understanding 

As we spiral through the essential questions of design — the elements — at increasing levels of specificity, we move 
through the phases of Define, Design, Demonstrate, Develop, and Deliver. 

Summary 

Recent trends suggest that instructional design models should reflect the way instructional practitioners 
implement design and remain responsive to the needs and constraints of the workplace. Consistent with these trends 
(Gustafson & Branch, 1997; Tessmer & Wedman, 1995), this model presents instructional design as a holistic. 
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collaborative process, acknowledges the concurrent nature of design practice, and supports the creation of scenarios, 
rapid prototypes, and other concrete means to move the design team forward. Three key ideas run throughout this 
view of instructional design: 

• Design involves conversations around critical issues (represented as five elements of design). 

• Design is an iterative knowledge-building cycle (represented as five phases of design). 

• Design is a collaborative activity among individuals representing different perspectives and 
expertise (represented as the ASC cycle). 

We believe this spiral model represents the iterative, knowledge-building manner in which instructional 
design is actually practiced. We accept the fact that different people may do the task differently, and in fact, there 
may be many, many ways of arriving at an effective product. Although there may be no one “right way” to design 
instruction, there are most likely themes that run through effective design efforts. We invite you to share stories and 
anecdotes of your design experiences so that as a profession, our models can better reflect the way we do our 
business. We see this model not as the “answer” but as the beginning and welcome your comments using the contact 
information provided on the front of this paper. 

Instructional design as a discipline has evolved over almost 60 years. The tools and techniques of practice 
continually adapt to new methods of delivering instruction and new developments in educational psychology. As 
research continues to yield information about how we actually practice our craft, our models will adapt to reflect that 
reality. As the field of instructional design continues to grow and respond to the needs of education and training, 
developments in delivery methods that demand our skills, and research and theory from educational psychology that 
inform our instructional strategies, we will remain a responsive, influential, and valuable contributor to society. 
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Current research and design of collaborative learning environments - often referred to as Computer 
Supported Collaborative Learning (cscl) Environments - tends to focus on surface level characteristics. Educational 
researchers and designers are busy, for example, determining optimal group size for problem-based education as 
opposed to project-centered learning. To determine optimal group size, students’ collaborative efforts and the results 
of these efforts are compared for groups of varying size in the different educational settings. This approach 
resembles comparative research on the use of different media in education that was strongly - and we had hoped 
definitively - criticized by Clark (1983). He eloquently argued that researchers tend to focus on the media used and 
surface characteristics of the education they provide. As a consequence, comparative research tends to be 
inconclusive and the learning materials developed tend to be unreliable at best and mathemathantic (from the Greek: 
mathema=learning + thanatos=death) at worst. This surface level approach disavows the fundamental differences 
between the real determinants of learning and behavior in education. 

A second problem is that educational institutions tend to apply traditional classroom ideas and pedagogy in 
non-contiguous collaborative learning environments, assuming that since these environments allow the interaction 
that we see in the classroom (e.g., chat, real-time meetings, shared applications) traditional pedagogy can be used. 
The proximate result is often disgruntled or disappointed students and instructors, motivation that is quickly 
extinguished, poorly used environments, wasted time and money, and showcase environments that are often not 
much more than computer assisted page turning. The ultimate result is the same as by the first problem, the death of 
learning. 

The solution is as simple as it is elegant, attending not only to technology, but also to the educational and 
social prerequisites for collaboration. This article provides a framework for designing such collaborative 
environments based upon the three prerequisites. It then goes into somewhat greater dept with respect to three non- 
surface level educational factors central to collaboration, namely task ownership, task character and task control. 
Einally, it presents empirical research on the affordances 

Affording Collaboration 

Instructional design is deterministic or causal in that it tends to focus on individual learning outcomes and 
tries to control instructional variables to create a learning environment that supports the acquisition of a specific skill 
(a certain person will acquire a specific skill through the implementation of a chosen learning method). With 
collaboration, the use of groups complicates this. Here, individual and group level variables affect the collaborative 
learning process such that it is nearly impossible to pre-define conditions of learning or instruction for a group 
setting such that interaction and skill acquisition are controlled. 

Instead of a classic causal view, design of collaborative settings requires a more probabilistic view on 
design. This distinction corresponds with the one made by Van Merrienboer and Kirschner (2001) between the 
‘World of Knowledge’ (outcome) and the ‘World of Learning’ (process). In the world of knowledge, designers 
construct methods by which given learning goals in a specific subject matter domain can be attained by the learner. 
In the world of learning, designers focus on methods supporting learning processes, and not so much attainment of 
pre-defined goals. 

This probabilistic view implies that more attention needs to be paid to learning and interaction processes. 
Due to the interactions between learners, each person in a group may acquire a given skill through the chosen 
method, but may equally likely acquire only a part of the skill or the skill and something unforeseen. It might even 
be the case that the chosen method is abandoned by the group and replaced by another, more idiosyncratic method 
for that group. The question is not what specific educational techniques and collaborative work forms cause, but 
rather what they actually afford, also often referred to as the affordances of a learning environment. 

What now are these affordances? Simply stated, they are the perceived and actual fundamental properties 
of a thing that determine how the thing could possibly be used (Norman, 1988, p. 9). Affordances are most visible in 
real life. Some door handles, for example, look like they should be pulled. Their shape leads our brains to believe 
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that is the best way to use them. Other handles look like they should be pushed, a feature often indicated by a bar 
spanning the width of the door or even a flat plate on the side. Gibson (1977) originally proposed the concept of 
affordances to refer to the relationship between an object's physical properties and the characteristics of an actor 
(user) that enables particular interactions between actor and object. "The affordance of anything is a specific 
combination of the properties of its substance and its surfaces with reference to an animal" (Gibson, 1977, p. 67). 
These properties/artifacts interact with potential users and may provide strong clues as to their operation. In our 
view, the concept of affordances offers an alternative framework for designing and evaluating iecles if appropriated 
to the educational context. 

Education is always a unique combination of technological, social, and educational contexts. Take typical 
classical education and group learning. The educational contexts are competitive vs. collaborative, the social 
contexts are individual vs. group, and the technological (physical) contexts are individual workspaces with minimal 
assortment of materials vs. group workspace with a rich assortment of materials. CSCL represents yet another 
learning situation. The educational context is collaborative, the social context is the group, and the technological 
context is a computer-mediated one. The Open University of the Netherlands, for example, uses a computer- 
mediated communication environment (technological) for competence -based learning grounded in social 
constructivism (educational) with minimal direct contact, maximal guided individual study, and primarily 
asynchronous, text based contact between students (social). 

When technology mediates the social and educational contexts such that their properties induce and allow 
specific learning behaviors we speak of ‘technology affording learning and education’ . This means that we must 
hold count with technological, social, and educational affordances. 

Technological Affordances 

Norman (1988) related perceived affordances to the design aspects of an object suggesting how it should be 
used: “Design is about [real and perceived affordances], but the perceived affordances are what determine usability” 
(Norman, 1998, p. 123). Norman links affordances to an object’s usability, and thus these affordances are designated 
technological or technology affordances (Gaver, 1991). Usability is a well known objective of industrial or product 
design dealing with physical objects ranging from video-recorders to teapots, and human-computer interaction (hci) 
dealing predominantly with graphical user interfaces composed of interface objects such as buttons and scrollbars. It 
is concerned with whether a system allows for the accomplishment of a set of tasks in an efficient and effective way 
that satisfies the user (e.g., Preece, Rogers, Sharp, Benyon, Holland, & Carey, 1994). It is not a single dimension, 
but deals with ease of learning, efficiency of use, memorability, error frequency and severity, and subjective 
satisfaction (Shneiderman, 1998). When creating such environments it is, therefore, important to consider these 
aspects. Otherwise we risk creating iecles that contain all the needed educational and social functionality (in 
Nielsen’s (1994) terms ‘utility’), but that cannot be handled by their users (i.e., the learners) because they are 
difficult to learn, access, and/or control in the same way video-recorders are. 

Social Affordances 

Kreijns, Kirschner, and Jochems (2002) define social affordances - analogous to technological affordances 
- as the “properties of a cscl environment that act as social-contextual facilitators relevant for the learner’s social 
interaction” (p. 13). Objects that are part of the environment can realize these properties; hence they are designated 
social affordances devices. When social affordances are perceptible, they invite learners to engage in activities that 
are in accordance with these affordances, that is, there is social interaction. 

In the ‘physical’ world, affordances abound for casual and inadvertent interactions. In the ‘virtual’ world, 
social affordances must be planned and must encompass two relationships. First, there must be a reciprocal 
relationship between group-members and the environment. The environment must fulfill the social intentions of 
members as soon as these intentions crop up while the social affordances must be meaningful and support or 
anticipate those social intentions. Second, there must be a perception-action coupling. Once a group-member 
becomes salient (perception), the social affordances will not only invite, but will also guide another member to 
initiate a communication episode (action) with the salient member. Salience depends upon factors such as 
expectations, focus of attention, and/or current context of the fellow member. 

Educational Affordances 

Kirschner (2002) defines educational affordances as those characteristics of an artifact that determine if and 
how a particular learning behavior could possibly be enacted within a given context. In other words, the chosen 
educational paradigm - the artifact - is instrumental in determining if and how individual and team learning can take 
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place. Educational affordances can be defined - analogous to social affordances - as the relationships between the 
properties of an educational intervention and the characteristics of the learners - in the case of environment the 
learners and the learning groups - that enable particular kinds of learning by them and in the environment for the 
other members of the group. 

Educational affordances in distributed learning groups encompass the same two relationships as social 
affordances. The environment must fulfill the learning intentions of the member as soon as these intentions crop up 
while the affordances must be meaningful and must support or anticipate the learning intentions of the group- 
member. Eurther, once a learning need becomes salient (perception), the educational affordances will not only invite 
but will also guide her/him to make use of a learning intervention to satisfy that need (action). The salience of the 
learning intervention may depend upon factors such as expectations, prior experiences, and/or focus of attention. In 
the next section we will discuss how these ideas can be incorporated into the design process. 

Design Guidelines 

According to Norman (1992), the major problem with most new technological devices and programs - and 
in our opinion also in their use in education - "is that they are badly conceived, developed solely with the goal of 
using technology. They ignore completely the human side, the needs and the abilities of people who will presumably 
use the devices" (p. 65). Take, for example, the escalator. This device, a moving stairway, was designed with the 
intention of speeding up human traffic in a stairway by increasing a person’s stair climbing speed with the speed of 
the escalator. As we all have probably noticed - with the exception of the British - is that it has actually achieved 
the opposite. Large crowds tend to gather and cue at the top or bottom of the escalator because the users tend to 
stand still on the escalator itself. This could be the result of human nature (i.e., inherent laziness) that was not 
entered into the design equation, the non-ergonomic step size that we encounter on almost all escalators (as we all 
have noticed when we have had to climb or descend an escalator that was standing still) or a combination of the two. 
Good use - and that means both usability and usefulness - requires a design process grounded in user-centered 
design research. 

We propose a Six-Stage Model for the design of environments on a general level (see Eigure 1). In this 
model, the designer must: 

1. Determine what learners actually do: Watch students interact, observe collaborating groups nteracting to 
solve problems, observe users interacting with software, et cetera, and do this before designing and 
developing. 

2. Determine what can be done to support those learners: Determine, based on stage 1, what actually needs to 
be supported / afforded and then proceed. 

3. Determine the constraints of the learner, learning situation and learning environment and the conventions 
that already exist: Look further than the technological constraints and conventions and take into account the 
educational and social constraints and conventions that play a role in collaborative environments. 

4. Determine how learners perceive and experience the support provided: There is a world of difference 
between (good) intentions and user perceptions thereof. Research and design must be carried out as 
iterative, interacting processes. New 'products' must be tried out with intended users at stages in their 
development where physical and conceptual changes can still be made. 

5. Determine how the learner actually uses the support provided: Analogous to stage 1, and following up the 
more formative evaluations carried out in stage 4 determine if the learner actually does what is hoped or 
expected. 

6. Determine what has been learnt: The goal of education is learning and there are three standards to 
determine the success of any instructional design, namely its effectiveness, its efficiency and the 
satisfaction of those learning and/or teaching. An increase in one or more of the standards without a 
concomitant decrease in any of the others means success. This is the proof of the pudding. 
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Learner / user 
experience 


What do learners actually do? 
What do learners want to do? 







How can we support what they do? 
What affordances are needed? 







Constraints / 
conventions 


What are the physical, logical and 
cultural limitations encountered? 







How does the learner perceive the 
support? 



How does the learner actually use 
the support? 



Learner / user 
perceptions 



Learner / user 
experience 



Support / 
affordances 



What has the learner / learning 
group actually achieved? 


Learning 







Figure 1 . Six-Stage Model of Design 

These six stages provide a general approach to instructional design of environments. But, this design needs 
to also assure that the type of social interaction regarded to be supportive for competency development actually 
occurs. Thus, complementary to the Six-Stage Model, a more specific, process-oriented design methodology is 
needed which supplies the designer with those questions which must be answered in Stage 3 of the general design 
level (i.e., to determine the constraints of the learner, learning situation and learning environment). Process-oriented 
methods may stimulate designers to adopt a probabilistic approach to CSCL design according to the expected 
interaction, paying attention to critical elements (constraints) affecting the interaction. Strijbos, Martens, and 
Jochems (in press), propose just such a methodology also consisting of six steps, namely: (1) determining the 
competencies to be developed, (2) determining the expected (changes in) interaction, (3) selecting the task type, (4) 
determining whether and how much pre-structuring support is needed, (5) determining the group size which will 
ensure that the type of social interaction needed for competency development develops, and (6) determining how 
computer support can be best applied. 

Space does not allow a comprehensive operationalization of the two frameworks for the whole of the 
educational process and all of the actors taking part in it. As most educational design centers around the task, the 
operationalization of the framework here will focus on Step 3 of the specific design level, i.e. selecting the task- type. 
We identify three constraints that need to be considered, namely task ownership, task character and task control. We 
regard task ownership, task character, and task control as defining factors in the educational affording of 
environments, which will be illustrated through specific prototypical design questions related to these factors. In this 
discussion we make use of those specific questions pertaining to the third stage of the general model in that they can 
be used to determine the constraints of the learning environment. 

Task Ownership 

Ownership in a group is influenced by two pedagogical principles, namely individual accountability and 
positive independence. Individual accountability (Slavin, 1980) was introduced to counter the free rider or 
hitchhiking effect: some students would not invest any (or only a minimum of) effort into group performance. By 
stressing individual accountability, what the group does as a whole becomes less important. It is perfectly valid that 
in a group environment, each group member is individually accountable for his or her own work. For example, in 
many problem based learning environments students’ sense of individual ownership is increased through grading 
them for their individual effort, irrespective of the group’s performance. Positive interdependence (Johnson, 1981) 
reflects the level to which group members are dependent upon each other for effective group performance (enhanced 
intra-group interaction). Positive interdependence holds that each individual can be held individually accountable for 
the work of the group and the group as a whole is responsible for the learning of each of the individual group 
members. Essential here is social cohesion and a heightened sense of ‘belonging’ to a group. Positive 
interdependence is evident when group members in a project-centered learning environment carry out different tasks 
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within a group project, all of which are needed in the final product. Interdependence can be stimulated through the 
tasks, resources, goals, rewards, roles or the environment itself (Brush, 1998). 

Positive interdependence, in turn, provides the context within which promotive interaction takes place. 
According to Johnson and Johnson (1996), promotive interaction “exists when individuals encourage and facilitate 
each other's efforts to complete tasks in order to reach the group's goals.” (p. 1028). In other words, individual 
accountability, positive interdependence, and promotive interaction counter the tendency towards hiding and 
anonymity. 



Task Character 

Traditional school tasks are highly constructed, oriented towards the individual, and are usually well- 
structured, short in length, well-defined, and designed to best fit the content and not reality. An archetypical problem 
is: “Two trains traveling in opposite directions ...” On the other side of the coin are real life (authentic) problems 
that are almost always ill-structured (Mitroff, Mason, & Bonoma, 1976) and/or wicked (introduced by Rittel & 
Weber, 1984; see also Conklin & Weil, 1997). They are often so complex and multifaceted that they can only be 
solved by multidisciplinary groups working together, where the group members assuage cognitive conflict, elaborate 
on each other’s contributions and co-construct shared representations and meaning. Examples of such wicked 
problems are the building a new type of automobile, the legalization of marijuana, and so further. 

Complex ill-structured problems require a different educational approach than simple, well-defined ones. 

To educate people to be able to solve these types of problems we need to choose for a whole task approach, since, 
after all, real life tasks are not neat segments of some idealized whole. Whole tasks need to be divisible into, non- 
trivial, authentic part-tasks, which aim at achieving ‘epistemic fluency’. Morrison and Collins (1996) define 
‘epistemic fluency’ as “the ability to identify and use different ways of knowing, to understand their different forms 
of expression and evaluation, and to take the perspective of others who are operating within a different epistemic 
framework” (p. 109). Ohlsson (1996) enumerates seven epistemic tasks that can be used as guiding principles for the 
design of collaborative environments (describing, explaining, predicting, arguing, critiquing, explicating, defining), 
since they indicate the ‘discourse -bound’ activities that learners will have to fulfill during collaborative learning and 
working. 



Task Control 

Task control is strongly related to "learner control" which has had a somewhat fluid and eclectic history. In 
its broadest sense, learner control is the degree to which a learner can direct his/her own learning experience (Shyu 
& Brown, 1992). More specifically, learner control is the degree to which individuals control the path, pace, and/or 
contingencies of instruction (Hannafin, 1984). New learning paradigms and new technologies expand this concept of 
control since they make it possible to provide learners with control over depth of study, range of content, number 
and type of delivery media, and time spent on learning. With these options, learners can tailor the learning 
experience to meet their specific needs and interests. For this reason, learner control is not "a unitary construct, but 
rather a collection of strategies that function in different ways depending upon what is being controlled by whom" 
(Ross & Morrison, 1989, p. 29). Indeed, learner control may be a continuum of instructional strategies in which the 
learner is provided with the option for controlling one or more of the parameters of the learning environment 
(Parsons, 1991). This control can be related to such aspects as context, content, sequencing, pacing, feedback, 
reinforcement and possibly even learning or presentation style. However, Reeves (1993) points out that a problem 
with researching learner control centers around what learner control really is, because it could be the pace of 
learning, as well as sequencing, content, and speed of a program; and much more. 

Task control in environments relates to a number of interacting aspects of the total environment that deal 
with determining the relevant set of actions that students can, should or must perform as well as what an adequate, 
applicable or best solution or solution path is. It relates to the roles of the teacher/coach versus those of the learners 
with respect to selecting the relevant activities and learning approach. An underlying assumption here is that learners 
are amply self-sufficient to be given control over their own learning activities and collaboration methods. Table 4 
provides questions relating to design decisions about task control (in terms of ‘pre-structuring’ needed) and the type 
of competence development that can be afforded. 
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Three Research Examples of Environments at the Open University of the Netherlands 

Functional Roles 

CSCL-environments at the Open University of the Netherlands are primarily text-based and asynchronous 
communication is used. Introducing this form of communication and technological tools introduces students to an 
unfamiliar realm. An approach to supporting such asynchronous coordination is the use of roles (Brush, 1998). Since 
roles promote group cohesion and responsibility (Forsyth, 1999; Mudrack & Farrell, 1995), they can be used to 
foster ‘positive interdependence’ and ‘individual accountability’ (task) ownership). The first research project 
discussed here focuses on the effect of functional roles that provide context independent process support - 
developed for higher education - on group interaction and specifically the effect on coordination. 

With respect to technological affordances it was decided to limit the technology to the primary process 
being facilitated: communication. In that respect e-mail was sufficient. Since a sense of belonging to a group is 
essential to team formation, roles provide a social affordance for the development of group cohesion and a sense of 
responsibility. Finally, the roles provide an educational affordance given the assumption that they decrease 
coordination in favor of task-focused communication. In addition, the functional roles used are context-independent 
and thus transferable to other content domains in which project-centered work is conducted. 

Method. 

Fifty-seven students enrolled in a course on ‘policy development’ and twenty-three students in a course on 
‘local government’ (49 male and 31 female). Age ranged from 23 to 67 years (Mean = 34.4, SD = 9.03). The study 
has a quasi-experimental random independent groups design and was conducted in a course on ‘policy 
development’ . The manipulation involved the introduction of a prescribed role-instruction in half of the groups (R- 
groups), aimed at promoting the coordination and organization of activities essential for the group project. The other 
half of the groups was completely self-reliant with respect to organization and coordination of their activities (NR- 
groups). Each group consisted of four students and throughout the course they communicated electronically by e- 
mail. Their task was to collaboratively write a policy report containing an advice regarding ‘reorganization of local 
administration’, a topical subject in the Netherlands (and a very ‘wicked problem’). For a thorough discussion of the 
results and methodological considerations of this project see Strijbos, Martens, Jochems and Broers (submitted) and 
Strijbos, Martens, Jochems and Prins (in preparation). 

Results. 

A non-directional Mann-Whitney test revealed no significant differences between conditions with respect 
to the group grade (Z = -1.549, df = 4, p > .05). Several scales in the self-report questionnaires comprised a single 
latent variable that was interpreted as ‘perceived group efficiency’ (PGE). An E-test for homogeneity of variances to 
investigate the hypothesis of equality of variances in both conditions for the model without random slope (E = 2.86, 
df = 4, p > .10) and the model with random slope (E = 5.86, df = 4, .05 < p < .10) revealed a tendency with respect to 
students’ awareness of group efficiency. The outcome of the content analysis corroborates this interpretation. 

A non-directional Mann-Whitney test revealed a significant difference with respect to ‘task social’ 
statements (Z = 2.121, df = 4, p < .05). Students in the role condition contributed more statements that expressed, 
either a positive or negative, evaluation or attitude in general, towards the group or towards an individual group 
member. Eurthermore, more ‘task content’ statements were observed in the role condition (Z = 1 .984, df = 4, 
p < .05). However, the assumption that this would be due a decrease in the amount of coordinative statements was 
not confirmed. In fact, a directional test revealed that the amount of coordinative statements in the role groups also 
increased (Z = 1.776, df = 4, p < .05; one-sided). Apparently, functional roles stimulated coordination and as a result 
‘task content’ statements increased as well (r = .73, p < .01). 

A comparison between both conditions of dropout during the course (‘quit’ category) reveals no 
differences. However, a comparison of the total number of students that did not finish the course (‘quit’ and ‘no 
credits’) shows a significant difference with respect to the distribution = 6,1 18, df = 1, p < .05). Eighteen 
students in the ‘non role’ condition - compared to eight students in the ‘role’ condition - failed to finish the course in 
time. 

Conclusion. 

The outcomes of this study clearly reveal that focusing solely on group performance outcomes (i.e., grades) 
as indications of successful collaboration is insubstantial evidence. As well as their design, the study of the 
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environment requires triangulation of research methods and different types of data to construct a representation of 
the collaborative process. 



Group Awareness Widgets 

The aim of the second research presented here is to create sociable environments that meet as much of the 
social (psychological) needs of learners through the explicit embedding of social functionality in them using the 
framework of social affordances. Within the area of human computer interaction and computer-supported 
collaborative work, researchers have already become aware that such sociable environments are needed by virtual 
groups (Donath, 1997; Feenberg, 1989). Bly, Harrison, and Irwin (1993) argue, “The smooth integration of casual 
and task-specific interactions, combined with the ability to meet informally as well as formally, is a critical aspect of 
productive group work” (p. 29). In contrast, they observe that “Most tools in computer-supported cooperative work 
(CSCW) are devoted to the computational support of task-specific activities, but support for cooperative work is not 
complete without considering all aspects of the work group process. When groups are geographically distributed, it 
is particularly important not to neglect the need for informal interactions, spontaneous conversations, and even 
general awareness of people and events at other sites” (p. 29). The kind of social affordances we wish to implement 
focus on stimulation of informal and casual conversations, stimulation of impromptu encounters, and bridging the 
‘time gap’ imposed by asynchronicity. All three aims imply proximity to be an important dimension of social 
affordances. The first two aims address proximity of place (i.e., spatial proximity) and the latter addresses proximity 
of time (i.e., temporal proximity) that can be bridged by using ‘traces’ which introduce a form of history awareness 
(Kreijns, Kirschner, & Jochems, 2003). 

In CSCL-environments, social affordances devices may be operationalized by group awareness widgets 
(GAWs). These widgets are software tools providing group awareness to provide virtual spatial proximity, history 
awareness, (i.e. the structured collection of all traces) to provide an overview of temporal proximities, and a set of 
communication media for perception-action coupling. 

Based upon the social affordances framework, we have implemented a first prototype of a GAW. This 
prototype is a first step, not all the implications of the framework are implemented; the set of communication media 
contains the traditional communication media (chat and e-mail) and pictures of the participants are used. 

online 

awareness 

hislorj' awareness 




t 

t 0 sec t - 1 1 days 

Figure 2. The prototype Group Awareness Widget 

The window has a time-axis along which strokes are displayed. The stroke length is an indication of the duration of 
the engagement. Each member is displayed with her or his strokes. Green (i.e., dark grey if printed) means on-line, 
red (i.e., also dark grey if printed, but a bit lighter as green) means off-line. Black means that there is no history data 
available for displaying. As can be seen from figure 2, history awareness is limited to 1 1 day’s; days a separated 
from each other by small vertical black strokes. Clicking on the portrait of a group member or on a stroke opens a 
dialog box from which the communication channels are accessible. The dialog box also contains personal and 
business information about that group member. 

As simple as this single segment of the GAW is, it may already invite the group learner to initiate a 
communication episode that is not based on the learning tasks to be done. It provides the group member with 
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information that may stimulate social interaction with others in the following ways: A group member may perceive 
that (s)he is not alone in the environment, even in the case that there are no group members currently online; group 
members may show up at regular times providing opportunities for future contacts in real-time; Patterns of ‘busy 
times’ become visible; A real-time conversation can be initiated with group members currently online (i.e., a chance 
encounter), or a message can be sent to a group member who is currently not online. 

In addition to the framework of social affordances, the second research also encompasses the concepts of 
sociability and social presence that have a relationship with the concept of social affordances. To this end, 
instruments are developed that measure the degree of sociability, social space, and social presence (Kreijns, 
Kirschner, Jochems, & van Buuren, 2003). Testing the hypotheses is the current focus of this research. 

A Formalism to Support Common Ground 

The third study discussed here aims at facilitating grounding in such environments (see Beers, Boshuizen, 

& Kirschner, 2003 for a more detailed description of this study). The main aim of our research is the design of 
social/technological affordance that invites and allows users to negotiate meaning while collaborating in context of 
solving wicked problems. This specific experiment was carried out to explore whether a negotiation widget or tool 
would positively affect the process of negotiating common ground. 

The process of negotiation starts when a team member makes her/his own, as yet unshared knowledge 
explicit or tangible to others. This can be oral, written, symbolic, et cetera. After one team member has made a 
contribution, others can try to understand it. In doing this they can consider aspects of the contributor (e.g., 
educational background, political views, et cetera) as well as their own beliefs and assumptions. A contribution is 
thus understood against the presumed perspective of the other, as well as against one’s own perspective (Bromme, 
2000). According to Clark and Shaefer (1989), the process of grounding continues until “the contributor and the 
partners mutually believe that the partners have understood what the contributor meant to a criterion sufficient for 
the current purpose” (p. 262). 

This exploration served as the basis for the design of educational affordances. CSCL environments often 
make use of external representations formed/restricted by a formalism: a set of objects and rules for making an 
external representation. In this research a formalism for facilitating negotiation of common ground in problem- 
solving groups was developed and tested. We expect the formalism to result in more negotiation activities and, 
ultimately, more common ground. The formalism was embedded in the newsgroup environment by adding specific 
message types, and specific rules about when one was allowed to post messages of certain types. 

Method. 

In a pilot study we studied people collaborating in a face-to-face setting who were required to making use 
of a pen-and-paper approximation of the negotiation formalism during their work. Participants were to solve a 
complex economics problem derived from a dynamic economics simulation game. First, the participants were 
allowed to practice with the simulation individually, and to solve the case individually, so that they were able to 
fully apply their own perspective to the task. Next, they solved the problem collaboratively, and after that 
individually again. All individual problem representations and solutions, as well as the group problem representation 
and solution, were gathered. The collaboration process was also videotaped. 

Procedure. 

Groups using the formalism were compared with groups not using the formalism. The formalism was built 
into a widget for a discussion list. A coding scheme for analysis of the collaboration process was developed in which 
different categories were used for new conversation topics (Contribution), verifying one’s own understanding of 
another’s utterance (Verification), clarifying the intended meaning of a previous utterance (Clarification), e.g. as a 
reaction to a verification, and Elaboration when talk continues about a certain topic, without verifying and clarifying 
intended meaning of utterances. A high number of clarifications and verifications was seen as indicative for explicit 
negotiation processes. To get an indication of common ground, overlap between individual problem representations 
after problem solving was determined. To do this, discussion topics were identified in the coding process to 
characterise the content of individual representations. Each topic was investigated with respect to whether an 
individual representation it was present after collaboration. The extent to which specific topics from the discussions 
were present in more individual representations after collaboration was seen as an indication for common ground. 
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Results. 



A preliminary analysis of the data is presented in Table 1. With regard to negotiation, results indicated that 
the formalism-groups spent more time on negotiation processes than the non-formalism groups (i.e., groups that 
used their own idiosyncratic representation method), as reflected by the number of utterances that were 
representative of negotiation. Furthermore, the formalism groups discussed more different topics than the 
idiosyncratic groups. Third, more members of the formalism groups participated in discussion (Participants per 
segment) than in the idiosyncratic groups. These results suggest a more equal representation of different 
perspectives in the collaboration process than in the idiosyncratic group. We conclude that the formalism seems to 
be able to affect negotiation processes by making them more explicit. 





Mean 




Standard deviation 


Formalism 


No formalism 


Formalism 


No formalism 


Time (s) 


3182 


2341 


390.58 


579.66 


Contribution 


25.00 


19.30 


4.36 


4.51 


Verification 


38.00 


19.30 


5.20 


8.50 


Clarification 


46.00 


26.70 


7.21 


6.66 


Elaboration 


118.00 


102.00 


34.51 


17.44 


Negotiation^ per contribution 


3.37 


2.38 


.09 


.58 


Participants per segment 


2.67 


2.47 


.09 


.07 



“ Negotiation = negotiation of meaning = sum of verifications and clarifications. 
Table 1. Differences in use of categories 



With regard to common ground, our results indicated that the formalism groups achieved more common 
ground than the idiosyncratic group (see Table 2). As mentioned, the formalism groups discussed more different 
topics, and captured more of these topics in their group external representation. Furthermore, members of the 
formalism groups mentioned more different discussion topics in their individual problem representations after the 
problem solving task, and there appeared to be more agreement between them than between the participants of the 
idiosyncratic groups (i.e., their individual representations resembled each other more closely than the individual 
representation made by the members of the idiosyncratic group). We conclude that the formalism indeed seems to 
have positively influenced the extent to which common ground has been negotiated. 





Mean 




Standard deviation 


Formalism 


No formalism 


Formalism 


No formalism 


Total number discussion topics... 


25.0 


19.3 


4.36 


4.51 


...in the group external representation 


19.3 


12.0 


9.50 


5.56 


...in the group solution 


8.7 


2.3 


2.52 


1.15 


...in one individual representation 


6.3 


6.3 


3.06 


2.08 


...in two individual representations 


4.0 


2.0 


3.00 


1.00 


...in three individual representations 


3.3 


2.3 


3.08 


2.31 



Table 2. Common ground 

These results have to be regarded with some caution. Only six groups were tested, and no statistical tests 
were used due to our small sample size. The results do provide encouragement to further develop the tool for 
facilitating the grounding process. 



Conclusions 

We have outlined a theoretical framework for the design of environments and illustrated this with examples 
from three research projects. The concept of affordances is central to this design and specifically in those cases 
where the learning environment centers on collaboration. With respect to the design of iecles it is not of primary 
importance what exactly is caused by different elements of the learning environment (learning is no longer causal or 
deterministic, but has become probabilistic). More important is whether the elements of a learning environment 
afford the type of competency development that was targeted. With respect to collaboration, the question is whether 
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the elements of the environment afford the emergence of that type of social interaction that is supportive for the 
acquisition of the targeted skill. 

But these questions cannot be easily answered. We, as designers often think that we know what our designs 
and products will do and how those, for whom our designs and products are intended, will use them. Unfortunately, 
this is not always the case. Each of the phases in the design process needs to be studied with respect to the specific 
choices that can and must be made. Some research is fundamental such as research to determine how iecle interface 
design or the way information is presented (e.g., Kester, Kirschner, & van Merrienboer, 2002) affects cognitive load 
and learning. Other research is more developmental in nature such as research on how group awareness widgets 
(Kreijns & Kirschner, 2002) affect the perception of sociability, social space and social presence. And still other 
research is applied such as how specific learners or learner groups perceive a specific iecle. 

Design of CSCL-environments needs to be carried out at two levels, namely a generic and specific level. 
The impacts of both levels have been illustrated with respect to task ownership, task character and task control. 
Clearly, the design of a CSCL-environment requires that both levels be taken into account with the specific level 
being a detailed depiction of the third stage of the general model in that it is used to determine the constraints of the 
learning environment. Although teachers and designers may prefer a clear set of design rules (i.e., first do A, then do 
B if you want to achieve C), a deterministic checklist with a limited number of categories is one step too far. We 
have, however, provided a number of specific design questions to stimulate teachers and designers to think more 
deeply about their instructional decisions, and not to simply rely on their traditional approach that “has worked so 
well”. 
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Use of Internet among Faculty Members of Anadolu University in Turkey 

Mestan Kiicuk 
T.C. Anadolu University 

Abstract 

Advances in information technologies have provided tremendous opportunities for professionals in almost 
all fields. For many, especially Internet along with its capability of providing easy and faster communications, 
access to resources, etc., has opened a new are in the history of world. 

Academicians are among those professionals who get benefits of Internet in their daily tasks. For instance, 
a faculty member who wants to conduct a research on any subject can reach so many up-to-date resources on the 
subject by couple of keystrokes and clicks. Once, before the spread of Internet use, accessing the up-to-date 
research finding was a big hassle. Faculty members use Internet not only for research purposes but also for 
instruction, publication, and communication. 

For the members of the universities in developing countries where they as well as other citizens of the 
country struggle with economic problems, Internet has become something inevitable. Now a faculty member by 
using Internet can discuss any topic with her/his colleagues all over the world, can publish results of her/his studies, 
conduct a research on information databases, and even offer a course to her/his distance students. 

The Internet and its ancestors (EARN - European Academic Research Network) have been available to Turkish 
academicians since the late 1980s. During the early days only a few people were able to use and access to these 
facilities. However, since the late 1990’s the number of the Turkish academicians who are using Internet everyday 
has burst. 

However, since there is no formal faculty development plans in Turkish universities and the rate of 
computer ownership at homes is around 3% among all the citizens of Turkey, questions such as “Do the Turkish 
academicians use Internet?”, “How do they, especially older ones, learn to use Internet?”, “What do they think of 
Internet?” are subject to curiosity. But, no study has been conducted to investigate patterns of Internet usage among 
Turkish academicians. 

The main purpose of this study is to investigate the patterns of Internet usage among faculty members of a 
university. In the study, it has been tried to answer following questions: 

(1) What is the frequency of academicians’ Internet usage? 

(2) How did they learn to use the Internet? 

(3) How often and for what purposes do they use Internet for? 

(4) Are they satisfied using the World Wide Web for research? 

(5) What is the self-efficacy level of academician’s for Internet? 

Method 

The design of the study was based upon the general survey method. The data were gathered through a 49 
items questionnaire. The first part of the questionnaire consisted of the questions related to demographic 
characteristics of the faculty members such as age, gender, academic titles, and departments in which they work. 
These data were used to compare if the Internet usage differ according to gender, age groups, academic titles, 
disciplines. 

The second part of the questionnaire involves questions about accessing a computer and Internet, frequency 
of using Internet, ways of learning to use Internet, frequency of using varying Internet tools (such as e-mail, 
newsgroups, WWW), purposes of using Internet, effective use of Internet tools and search engines. 

In addition two scales have been located at the last part of the questionnaire. The first one is a translation of 
the End User Computing Satisfaction Instrument (EUCS). The EUCS was developed by William J. Doll and 
Gholamreza Torkzadeh to test the user satisfaction from the Internet resources. This scale includes 15 questions to 
measure five factors; content, accuracy, ease of use, format, and timeliness. Each item was answered on a Likert- 
type scale of 1 (almost never) to 5 (almost always). The mean for each question was assumed to be 3 (sometimes). 

The second scale is a part of the Online Technologies Self-efficacy Scale developed by Marios Miltiadou. 
The original scale consisted of 30 items. Eirst 10 items of this scale are related to the self-efficacy of the users for 
general Internet competencies. These 10 items were included into the questionnaire. In this scale each question was 
answered 1 (not confident at all) to 4 (very confident). 
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Descriptive statistics, variance analysis and t-test were used to analyze the data gathered through the 
questionnaire. 



Results and Discussion 

The questionnaire was administered to the faculty members working in different colleges and departments 
of Anadolu University, Turkey. At the end of a stratified random sampling, 295 academicians were selected out of 
982. Paper and pencil type of the questionnaires were distributed all sampling and allowed one month to send back. 
The 248 (%84) academicians filled out the questionnaire. Academic departments of respondent are shown in Table 
1 . 



Table 1 Academic Department of Respondents 



Departments 


N 


% 


Faculty of Education 


44 


18 


Faculty of Engineering and 
Architecture 


38 


15 


Faculty of Open Education 


36 


15 


Science Faculty 


35 


14 


Faculty of Communication 


19 


8 


Science 




Faculty of Fiterature 


19 


8 


Faculty of Pharmacy 


19 


8 


Faculty of Fine Arts 


18 


7 


Faculty of Economic and 


11 


A 


Administrative Science 




Faculty of Faw 


9 


4 


Total 


248 


100 



Respondents were asked to select from five categories of faculty rank: research assistant, instructor, 
assistant professor, associate professor, and professor. 69 (28%) respondents were research assistant, 70 (28%) 
instructor, 62 (25%) assistant professor, 32 (13%) associate professor, 15 (6%) professor. 

Results have shown that Internet is used every day by the majority of faculty members and the majority of 
them have used internet over three years. Many of them (86%) learned to use Internet through independent learning 
method (trial and error). Among the Internet tools, e-mail (M=4.73) and World Wide Web (M=3.79) are the ones 
most frequently used by academicians. Respectfully other results are electronic journals; document delivery 
services; external catalogs and bibliographies; discussion lists and news groups; and software download. 

Each part of the questionnaire, t-test and one way ANOVA was run to determine if significant differences 
existed among the independent variables of gender, age, academic titles, and disciplines. Significant differences 
were found for the discipline factor in terms of e-mail (p=.037), electronic journals (p=.001), and document delivery 
services (p=.001). In all cases means for science were higher than means for social science. 

Significant differences were found for the age factor regarding to e-mail (p=.040; M=4.73; sd=.66), discussion list 
and news group (p=.048; M=2.83; sd=1.19). In other cases weren’t found any significant differences. 

Academicians use the Internet and WWW every day for research purpose (M=4.55). Respectively the 
other mean scores are communication (M=3.83), teaching (M=3.05), and publication (M=2.59). According to these 
results academicians often use Internet and WWW for communication purpose, sometimes for teaching purpose, and 
uncommonly for publication purpose. Significant differences were found for the age factor and the gender factor in 
terms of research purpose. 

The study also revealed that the faculty members are trust themselves about using the Internet, because 
each item mean scores are higher than critical level (Md = 2.51). Also faculty member are self-confident about the 
use information retrieval resources. 

They use search engines every time (M=4.39) for finding information and often use subject directories 
(M=3.75), data bases (3.73), and electronic journals (M=3.72). On the other hand, although faculty members mostly 
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use search engine to retrieve electronic information, generally they don’t use search operator that simplify to retrieve 
electronic information. 

In addition, each respondent was asked to complete the EUCS survey by indicating how often they would 
respond to each of 15 question on a scale (from l=almost never to 5=almost always). These questions were grouped 
into six factor section, and the sum of the responses in each section determined individual factor scores. The factor 
scores were totaled for overall scores. Assuming an expected mean of 3 (sometimes) for each question, the means 
returned were significantly higher than the expected means. 

T test were run for all individual question scores, factor scores, and overall scores, using the expected 
means as the comparison. All scores showed a significant differences at p = .05. 

One way ANOVA was run to determine whether any significant differences existed among the independent 
variables of gender, age, academic titles, and disciplines. 

Significant differences were found for the academic titles factor in terms of content (p=.014), accuracy 
(p=.005), timeliness (p= .019) factors and overall EUCS (p= .014) scores. Also significant differences were found 
for the gender factor in terms of accuracy (p= .045). These finding shows that academicians question the content and 
the accuracy of most of the web-based information. 
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Web-Enhancing a Traditional Class: Lessons Learned and Lessons That Need 

to Be Learned 

Carmen L. Lamboy 
Ana G. Mendez University System 



Rationale 

Business Administration students need to acquire skills that will enable them to produce effective written 
business communication. Effective English communication is an asset for graduates since the majority of 
employment opportunities aim at bilingual candidates in Puerto Rico. 

In today’s world, these students also need to enhance their computer and technology skills. The usage of 
the Internet and technology in Business is a reality and it is here to stay. This paper intends to share the different 
activities that can be used to practice and enhance English and technological computer skills. Some results will be 
shared. 

Before going on-line: A study of the Needs of Corporate World and the Students 

The research place had carried out a needs assessment in the year 2001. Corporations and student focus 
groups were interviewed. Some results follow. 

Corporate World 

Nine corporations were interviewed. These corporations represented the following areas: 
tourism/hospitality, health services, technology, business administration, technology, and education. 

In every case, the corporate responses to the needs were that they wanted Hispanic graduates who have truly 
comprehensive bilingual skills. Being Bilingual and being able to use technology were valued skills and was seen as 
a real contribution to the workforce. All interviewees pointed out that English language deficiencies - particularly 
bilingual deficiencies in areas of professional training — were holding back the advancement of many Hispanics in 
their organizations 

The students’ perspective 

A focus group was carried out. The subjects of the focus group were divided into four groups of 7-10 
participants each, in the 18-49 age group, male and female presumably, who have attempted at least 24 college 
credits but did not earn a college degree. 

The goals of the study were to determine potential students' educational desires, and which elements of the 
proposed web-enhanced course are found favorable and which are not. Subjects stated what they wanted was: 
Language support in terms of bilingual instructors. 

1 . An exciting, innovative, dynamic approach, with small groups 

2. Job relevance in terms of instructors who have been in the field and curriculum that is applicable to the job. 

3. Internet, online capabilities. Comments included having access to information online, hosted by the school, 

or if the school had online workstations that the student could use. 

Eor some participants the lack of English competence had prevented them from succeeding academically in 

past attempts. 

As to why they wanted to pursue a degree, some of the people in the focus group had the experience that 
they had to prove that they were capable of doing a job. This may have to do with overcoming prejudiced attitudes, 
but it may have to do with their command of English, either in correctness or proper business use. 

Now as the teacher that read this report, I needed to ask myself several questions before going on-line and 
starting to plan on web-enhancing my courses. Some of the questions were: 

1 . How many students have computers or computer access? 

2. How much do students know about navigating the web? 

3. Do you trust your students? How do you know if students are really doing the work and not some 
expert? 

4. How well prepared is the university to offer students the support they need? 
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5. How well prepared is the university to offer you the support you need? Are you ready to be a 24/7 
teacher? (Literally) 

Lessons Learned 

1. Communication cannot be overstressed! Having clear expectations, course requirements and providing 
clear direction is important for the success of the group and the class. This includes a clear idea of 
coursework, technical requirements and skills. It also has to include clear evaluation standards. 

2. Posting Course Documents. Be clear when preparing the organizational structure. Be creative when 
preparing main course content. Attach documents, handouts, images and media in a timely fashion. 
Customize course structure with clear delimited folders. 

3. When posting course-related assignments, be sure to include clear evaluation criteria, due dates, and clear 
instructions. 

4. What can we do with Quiz results? When we met face-to-face, we would center interaction on these 
results. We would work in smaller groups with students who show similar difficulties. It was easier to 
individualize instruction. It also meant that I needed to analyze quiz results and make sure I planned 
appropriately. 

5. Use all available tools for communication. (Discussion board. E-mail, Student Roster, Student Pages, 
Group Pages, Chat) 

6. Providing external links for students was a great way to add online resources to a course. These links 
provided students with access to global resources. 

7. Provide a place where students can check their grades, drop off their work, and edit their information and 
student page. 

8. To convince stakeholders, administrators and myself, keeping track of statistical information about course 
usage, such as the frequency that students visit course components was necessary. When an instructor can 
see which pages and links are being used and track peak and off-peak access times it tells them about their 
students and if the materials developed are being accessed. 



Advantages and Disadvantages 



Advantages 


Disadvantages 


Students are not afraid to 
participate. 


More time must be dedicated (teacher) 


Students have time to process data 


Feedback is not immediate 


Self-paced 


Can become too technical. 


Students dedicate more time to course 

Individual attention 

Students develop technological skills 

Student centered 


Technical problems and difficulties could hinder 
process. 



My own learning 

I definitely acquired many new technical skills as I started looking for different technologies I wanted to 
incorporate in my course. I practiced my patience. Many times dealing with different assignments from different 
people at different times. Answering some of the questions more than once and dealing with deadlines was 
challenging. I learned that flexibility was ok, as long as it didn’t become a habit. I had to revisit the teaching 
strategies that I used in a face-to-face environment and adapt them to on-line learning. Definitely it also re-affirmed 
that students like to interact among themselves and learn from each other. I most importantly there was NO 
SIGNIFICANT DIFFERENCE between students learning at a distance and students learning in a brick and mortar 
classroom. The future holds many promises, and I believe that on-line learning will not totally substitute face-to- 
face learning, but that teaching and learning will be enriched as new and diverse technologies emerge and old 
technologies enhanced. 
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Visualizing Knowledge: Designing Information Graphics for Instruction 

Vincent E. Lasnik 

Independent Instructional Technology Consultant 

Abstract 

Illustrations, graphs, charts, diagrams, photos and other visual images are common components of 
technical, creative, and educational communications to convey conceptual, statistical, procedural, qualitative, 
quantitative, static and dynamic information. But how do we determine the optimal standards and attributes for 
effectively designing and using graphics for visualizing knowledge in particular situations? This paper describes 
the grounding principles of information design for instruction and the continuing development of prescriptive visual 
taxonomies for learning, training and development. 

“Graphics is the visual means of resolving logical problems.” 

J. Bertin(1981,p. 16). 

This paper discusses descriptive concepts and prescriptive principles of visual perception and learner 
cognition that are emerging in the design and development of effective instructional course materials. Suggested 
strategies for the successful integration of high-quality instructional media within effective pedagogical frameworks 
for student course implementation activities is also briefly discussed. 

Effectively illustrated courseware is systematically developed using a synthesis of (a) information design, 
(b) media design, and (c) interactivity design subsumed within a unifying foundation of (d) the first principles of 
instructional design. This will require the insightful fusion of this synergetic tetrad enabling learners to construct 
new knowledge through hands-on, learn-by-doing experiences. Using integrated, interactive goal-based scenarios, a 
student might follow open-structured, collaborative role-playing investigating problems and cause-effect 
relationships. Effectively designed data representations and interpersonal transactions (including text, streamed 
audio, static and animated graphs, charts, technical illustrations and digital images; photographic and streamed 
motion video, student-teacher interaction, self-assessment problems, Internet hyperlinks, listservs, threaded peer-to- 
peer discussion boards, database-augmented hybrid DVDs) will complement these intrinsically-motivating learner 
activities. 

One of the central problems and corresponding challenges facing the multidisciplinary fields of educational 
communications and instructional technology has been in the construction of effective taxonomies for visual 
communication (Horn, 1998; Tufte, 1997; Ware, 2000). Numerous research studies and encyclopedic volumes have 
contributed to our evolving understanding of the core principles underlying sound methods and strategies for 
appropriately representing data information in particular contexts (Card, Mackinlay, & Shneiderman, 1999; 

Eleming, 1987; Harris, 1996; Wildbur & Burke, 1998). The compelling task of creating comprehensive conceptual 
models of complex phenomena that are parsimonious and robust without being overly reductionist — and concise and 
cogent without being over-simplifications — continues to be formidable. A re-examination of the fundamental issues 
in designing epitomes of visual communication might positively contribute to this collaborative endeavor. 

Graphic Theory: Envisioning the Problem Space 

An explicit orienting task which requires the learner to represent a problem situation graphically can act as 
a cognitive strategy activator appropriate to many content domains (Derry, 1985; Eleming, 1987; Rigney, 1978). 

The mnemonic technique of using visual imagery in the form of graphic illustrations has been shown to assist 
learners in associating coordinate concepts within a logical framework and thus significantly improve concept 
attainment (Dwyer, 1978). Also, many of the critical heuristic methods for perceiving, representing, analyzing and 
ultimately solving a problem can be augmented by the drawing of a graph or diagram (Polya, 1957). A well- 
constructed graph, illustration or diagram can clearly and unambiguously represent all of the important concept 
relationships in a set of data or in a problem situation. Moreover, Bertin (1981) emphasized that: 

“In considering hypotheses and methods, it is necessary to envisage the whole problem. The matrix analysis of a 
problem is a process which enables us to see the whole, that is, to construct it graphically, and to "foresee" the 
possible choices and their repercussions. “(p.l7). 

Therefore, the use of graphing as both a heuristic method and as a conscious cognitive strategy could 
enhance the problem solving repertoire of learners and contribute to their ability to (a) systematically observe. 
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decompose and define the problem in an X-Y matrix system; and (b) process, interpret and verify the hypothesized 
relationships between the data (Mayer, 1983; Nickerson, Perkins, & Smith, 1985). 

Problem-Based Visualization for Learning 

Problem-oriented techniques have been defined by many labels from inductive to discovery. Ausubel, 
well-known for his coinage of the “advance organizer” concept, interpreted the problem/rule duality to be 
comprised of discovery versus reception processes. For Ausubel (1963), receptive learning consisted of presenting 
to the student the entire, completely defined content of what is to be learned in final form. Discovery was a more 
omnibus category that included problem-solving and inductive reasoning. Collectively, these approaches entail an 
underlying holistic pedagogy of inquiry, guided discovery, and reflective thinking. Moreover, problem-oriented 
techniques, such as simulations, can use the advanced technical capability of computer technology to apply the 
cognitive theories of generative learning and individualized, intrinsically motivating instruction (Keller, 1983; 
Wittrock, 1974, 1978). Unfortunately even to this day, much of what constitutes secondary and college-level 
instruction continues to be predominantly receptive learning approaches that disembody information into 
predigested abstract chunks that are easy to deliver but provide little scaffolding for meaningful learning (Mayer, 
1989, 1999). This trend is also observed in the promulgation of factoids and blather lines in commercial mass 
communication and collaterally by the poor design and use of sound visualization techniques in schools. 

The Roots of the New Paradigm in Instruction 

Constructivist theory and practice have now earnestly captured the imagination of the academic world 
(Reigeluth, 1999). The dominant intellectual scaffold for the emergence of constructivist theory was clearly the 
cognitive movement in instruction. Cognitive approaches (which emerged during the late 1960s and matured into 
the 1970s and 1980s) are concerned with the active, generative construction of meaningful knowledge in the mind of 
the learner. Serious future research into any aspect of the dynamic instruction and learning process must maintain 
an awareness that instruction is mediated by student thought processes. The literature on cognitive models has long 
posited that the fundamental characteristic of the mind as a cognitive system is adaptability: the long-term ability to 
modify existing internal conceptual schemata to absorb new information (Eysenck, 1984; Glaser, 1977; Wang, 1980; 
Wittrock, 1978). Learners construct meaning as they approach new content domains and problem-solving tasks 
through active processes of inquiry, exploration, and hypothesis-generation and testing (Taba, Durkin, Fraenkel, & 
McNaughton, 1971; Wittrock, 1974). Thinking critically and formulating generalizations and principles through the 
manipulation and observation of data is a concept formation activity which places the major responsibility for 
learning on the student (Taba, et ah, 1971; White, 1969). Generally, these studies emphasize that thinking involves 
an active transaction between an individual and the data which he or she is studying. Data becomes meaningful only 
when an individual performs certain cognitive operations upon such data. Moreover, all students are viewed as 
potentially capable of thinking at abstract, analytical levels under the appropriate conditions of learning, and 
instructional strategies can be designed which will improve the higher-order thinking skills of individual learners 
(diSessa, 1977; Kirby, 1984; Nickerson, et ak, 1985; Perkins, 1986; Shuell, 1986; Wittrock, 1978, 1974). 

Cognitive strategies have been prominent in the development of instructional design and learning theory 
(Briggs, 1977; Dansereau, 1978; Gagne & Briggs, 1974; Grippin & Peters, 1984; Rigney, 1978). Appropriate 
cognitive strategies can facilitate the learner in "learning how to learn, how to remember, how to carry out the 
reflective and analytic thought that leads to more learning." (Gagne, 1977, p. 167). Moreover, learning a model of 
the underlying reasoning process used to generate solutions to problems within a domain is an important aspect of 
formalizing knowledge and in fostering skills in concept acquisition (Polya, 1957). A strategy utilizing such a 
model can become a conscious "metastrategy" used by learners as a powerful tool in their approach to understanding 
novel problem situations and concepts within the content domain (Allen & Merrill, 1985; Derry, 1985; Mayer, 

1989). 

The terms "concept" and "principle" have often been used broadly and interchangeably. Reigeluth (1983) 
distinguishes between concepts, which are predefined categorizations of phenomena, and principles, which are 
observed as relationships of change. These changes may simply be correlational, where two or more elements 
change simultaneously, or they may be causal, where a change in one element causes a corresponding change in 
another. While a concept consists of "a set of objects, events, or symbols that have certain characteristics in 
common," principles describe "the relationship between a change in one thing and a change in something else." 
(p.343). 



More recently Eisner (1998) — a strong proponent of visual literacy and arts education in the schools — has 
advocated the value of expanding our design and implementation of curricula that emphasize visualization skills for 
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expression, communication, and learning generally. He has posited a set of underlying premises that support the 
case for cognitive and social constructivism including the observations that (a): the form of representation influences 
both the processes and products of thinking; (b) different forms of representation develop different cognitive skills 
and forms of thinking; (c) the selection of a representation form influences not only what can be represented, but 
also what can be envisioned; (d) each form of representation can be used in different ways; and (e) different forms 
of representation can be combined to enrich the array of resources students can respond to. 
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Figure 1. Apollo 1 1 Lunar Mission Single Hour Timeline Flight Plan for July 1969 (Source: Historic mission data 
from public domain Website at http://www.NASA.gov) 



Building a Better Visual Metaphor 

The question of identifying the most efficient forms of graphic representation for instruction demands we 
go deeper into fundamental questions of what educational technology and active learning is really all about. The 
focus should be toward fully exploiting the uniqueness, accessibility, controllability, interoperability and 
extensibility of diverse media (Brown & Chignell, 1995; Kristof & Satran, 1995). Importantly, we must critically 
examine underlying assumptions of what teaching and learning must become to actualize human potential. The 
really difficult task is reconceptualizing the technology “to build a better metaphor” — one which emphasizes the 
experiential, active construction of cognitive strategies and robust mental models (i.e., metamodels) in the learner 
(Mayer, 1999). 

Parsimonious Graphic Design for Instruction 

Intuitively, good design aesthetics should adhere to minimalist information design guidelines and (a) 
reduce cognitive and perceptual load by minimizing superfluous cosmetic elements when designing image and 
screen “real estate” while (b) maximizing the value of data content using clear, cogent text, graphical representations 
distilled and re-edited for high signal to noise ratios, and complementary, reinforcing and non-gratuitous media 
design (Carroll, 1998; Doumont, 2003a & 2003b; Hansen, 1999; Lowe, 1993; Tufte, 1990). Tufte (1997) has 
identified this “less is more” characteristic as the smallest effective difference. Figure 2 shows a clean example of an 
instructional graph applying the principle of parsimony. 
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Figure 2. Sensitivity to Precipitation Bar Graph (Source: Pat Grimaldi, 2D/3D Illustrator) 



Mijksenaar (1997) has synthesized a simple taxonomy to consider when designing communication graphics 
that is structured around three organizing categories of visual function: 



Table 1. A Basic Graphical Variable Organizing Scheme (Mijksenaar, 1997) 



Functional Categories 


Visual Elements 


Distinguishing 


color 


classifies according to 


illustrations 


category & kind 


column width 
typeface 


Hierarchical 


sequential position (chronology) 


classifies according to 


position on page/screen 


importance 


type size 

line & bar spacing 


Supporting 


areas of color & shading 


accentuating & emphasizing 


lines & boxes (e.g. stroke) 
symbols, logos, icons 
text attributes (e.g., bold) 



Adding Layers of Visual Learning Value Using New Media 

Well-designed learner-centered secondary science courseware would require students to learn concepts and 
change relationships empirically by manipulating 2D and 3D objects and visually-represented variables within a 
plethora of temporal, spatial, and parametric simulations and multivariate procedural models and animations 
fostering scientific visualization (Fraser, 2001). After complete operational prototypes of comprehensive graphics 
have been produced, augment and enhance collateral and concurrent course instructional media (resources 
permitting). For example, if feasible add knowledge objects such as manipulable 2D and 3D static graphics and 
dynamic animations, interactive multidimensional and cross-disciplinary timelines (with synchronous database 
cross-references between domains), immersive goal-based simulations to illustrate concepts, principles, change 
relationships, tradeoffs, decision-making operationally, interactively and experientially (See Figure 3). 
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Figure 3. Mayan City Reconstruction in both 2-D and 3-D Representations (Source: Pat Grimaldi, 3D Illustrator) 



The Pedagogical Power of Interactive Graphics 

These objects should ostensibly be as learner-controlled as feasible. Visual displays should also be made 
more learner-accessible by producing indexed galleries with thumbnail images organized by topic/concept. In 
addition, juxtaposed simultaneous comparisons of the same image data or a close variant can provide a visual 
synergy that affords learners the opportunity to develop layers of understanding about the same phenomenon as 
shown in Figures 3, 4 and 6 (See also “multiples” and “confections” as described in Tufte, 1990). 
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Figure 4. Comparison of Cranial Structures (Sources; left; Visible Human Project; right; MRI database. Both 
images downloaded from the U.S. National Library of Medicine’s public domain Website at 
http;//www.nlm.nih.gov). 



Use of Visually-Engaging Simulations for Learning 

The pedagogical strategy of visually-based simulation (See Figure 5) as a pedagogical technique facilitates 
the learner to (a) generate and test hypotheses about the behavior of a system by manipulating certain elements in 
the system model while keeping others constant, (b) distinguish cause and effect relationships in the system, (c) 
distinguish between facts and hypotheses and between relevant and irrelevant elements, (d) identify previously 
unclear interdependencies and associations within the system, and (e) develop adaptive decision-making and 
higher-order thinking skills (Allessi & Trollip, 1985; Crawford, 1966; Cunningham, 1984; Greenblat, 1975; 
Thiagarajan & Stolovich, 1978). 




Figure 5. DNA Blaster Genetic Modification Procedural Animation (Source; Pat Grimaldi, 2D/3D Illustrator) 



Finally, high-bandwidth Internet technologies can provide powerful heuristic scaffolding through content- 
enriching hyperlinks to constantly updated public Website databases (e.g., government agencies like NASA that can 
compare dynamic time-synchronized visualizations) to facilitate student research and student thought-question 
follow-ups for time-based data and information-rich resources (See Figure 6). 
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Figure 6. Synchronous Time-Based Dynamic Data Comparisons (Source: 2002: Geo-thermal & geo-forestation 
satellite data visualizations from public domain Website at http://www.earthobservatory.NASA.gov) 



Integrating Content, Method, Function & Visual Form 

As good teachers know, much of the content is actually the method. This is particularly relevant to 
incidental learning, collateral learning, creative and practical performance skills, heuristic and procedural 
knowledge, and problem-solving transfer capabilities long after the immediate content of the current lesson has been 
forgotten. To this end, efficiencies in the field of instructional technology will require a design synthesis towards 
the deliverable goal of fully-integrated media built within a research-based, theory-grounded, learner-centered, 
problem-oriented focal architecture based on the active acquisition of real, transferable skills and metastrategies, and 
authentically-assessed performance competencies. Optimal instructional graphics are thus intentionally designed to 
synthesize content and method, utilizing multiple modalities (a) to augment skill acquisition predicated on learning 
by doing; (b) bridging individual differences in learning style, preference, motivation, proclivity and capability; (c) 
integrating these differences with content domain knowledge driven by the glue of an outcomes-based, problem- 
oriented, goal-based, performance assessment of competency (i.e., mastery); and (d) providing a coherent, logically- 
consistent, robust systems design (i.e., recursively in terms of production process and instructional product, 
treatment intervention, and evaluation) to develop and implement a unified learning environment. 



Summary 

In summary, a number of prescriptive strategies for improving the visualization of knowledge can be 
proposed: 

(1) Thoroughly organize course into macro-level and micro-level components. Use structured visual tools 
like outlines, timelines, flowcharts, both table-of-contents and index type lists, thematic logically-presented menus 
and concept maps to unambiguously convey this navigational structure to the student and assist course participation 
and wayfinding. For each course level, use advanced organizers with broader learning objectives, core topics, 
themes, concepts, and learner performance objectives clearly described in unambiguous language indicating 
measurable outcomes, assessment techniques, and demonstrable artifacts of competency. 

(2) Embed illustrative case studies and relevant anecdotes within lecture scripts; embed associated student 
activities for deep, connected learning and reflection (i.e., not mere “note-taking” but reflective thinking higher- 
order skills requiring interpretation, analysis, evaluation, decision-making, weighing alternatives, projecting 
outcomes, proposing solutions, formulating original questions, problems, and causal hypotheses, creatively 
interconnecting ideas, juxtaposing examples and counter-examples) like first-person writing assignments (e.g., 
journaling), “action research” papers, local follow-up investigations, peer sharing (including peer tutoring, peer 
review/evaluation) related threaded discussion topics and student self-assessment “thought questions” within each 
module to enhance student interest, reflective thinking, connection to material and active/participative learning. 
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(3) Concrete concepts and key principles (e.g., change relationships and multivariate, co-dependent inter- 
relationships) should be conveyed via simple process animations; DVD & Web-based versions of these animations 
(using tools like Flash, QuickTime, etc.) and small streaming media demonstrations should be made available as 
learner-controlled (at a relatively discreet, disaggregated, short-duration, fine-tuned level of segment granularity), 
interactive time & motion process and parametric simulations, slideshows, multiple-path walk-throughs. Add non- 
lecture, on-location media components whenever feasible, including one-on-one interviews, peripatetic walk and 
talk slideshows; focusing on object detail close-ups in conjunction with case studies and problem examples as 
appropriate to enhance narrative interest and dovetail visual concepts; utilize appropriate reinforcement of real 
images or illustrations with abstract concepts and principles demonstrating change relationships. 

(4) Reduce cognitive load by reconceptualizing aggregate and superordinate concepts into smaller 
information “chunks” (i.e., subconcepts, attributes and component elements). Modular course content can be further 
granularized into sub-levels (e.g., segments, mini-lectures) as feasible; again, content elaboration can be either Web- 
supported (e.g., outside hyperlinks, HTML/DHTML page text & images, slideshows, hyperlinked & annotated 
PDFs) or from other sources (e.g., hardcopy course reading materials). 

(5) Re-conceptualize instructional systems models through the use of powerful visual metaphors like the 
example suggested in Table 2; 

Table 2; Use of Conceptual Metaphors in Representing Knowledge 



Building Architecture 


Knowledge Architecture 


buildings 


courses, scope & sequence (curricula) 


macro-design approaches 


instructional design approaches 


purpose & function of building 


information design 


properties of materials 


media design 


patterns of interior/exterior space 


interactivity design 


structure lifecycle (repair, improvement) 


iterative courseware design 


settlements, zones, cities 


lesson activities, modules, units 


power, water, air, transportation 


courseware/online infrastructure 



Conclusion 

The unfulfilled promise of visualizing knowledge through effective information graphics will need to 
involve a pragmatic framework for (1) applying research-based theory and practice grounded in the fundamental 
first principles of an eclectic set of cross-disciplinary “best practices” from fields like instructional design, technical 
communications, information design, interactivity design, visual design, and media design while (2) fostering the 
systematic implementation of innovative, outcomes-based, problem-oriented, learner-empowering pedagogies 
capitalizing on current and emerging communication capabilities. We cannot separate content from method in 
instruction, nor educational philosophy from pedagogy — much as the courseware management and commercial 
online learning industries might like to disagree. Real learning will have to translate to real skills and extensible 
abilities (including the divergent-thinking ability to learn continuously on-the-job, develop analytic, heuristic and 
creative transfer skills and resolve novel problems operationally), and thoughtfully-designed information graphics 
can make an important contribution to this learning. William Blake’s inspirational aphorism summarizes our 
ongoing search for a comprehensive visual language to communicate, to educate, and ultimately to create: 

When the doors of perception are cleansed 
Things will appear as they truly are. 
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Abstract 

This paper describes preliminary findings regarding how seemingly nonsensical conversations in a 
synchronous Book Talk among elementary reading teachers might, in fact, support the development of a community 
of practice. Using Burnett’s (2000) typology of exchanges for virtual communities, noninteractive and interactive 
online behaviors, as evidenced in the text -based ‘talk’, are explored. An analysis of conversation showed evidence 
of mutual engagement, joint enterprise and the development of shared repertoires, three activities related to the 
maintenance of coherence in communities. 



Introduction 

Today's prevalent practice of online chats and discussion forums through the World Wide Web 
dramatically transforms the technology landscape in relation to human behaviors. The widely used Internet gives 
rise to a social interaction phenomenon in which participants 'converse' using electronic textual messages as their 
means of communication. These emergent computer-mediated environments are more than merely as a source of 
information, entertainment, and basic communication. In some respects, the Internet becomes an alternative avenue 
for people's self-expression, stress release, professional development, search of identity, socialization, and social 
learning etc. (Burnett & Bonnici, 2003; Turkic, 1995). 

By definition, online communities are "designed communities using current networked technology, 
whereas communities of practice emerge within the designed community via the ways their participants use the 
designed community" (Johnson, 2001, p. 45). Within the context of this definition, the term online communities 
covers the broad spectrum of both the synchronous and asynchronous forums such as web-based newsgroups, 
discussion boards, listservs, conferences, chat rooms and so forth. Regardless of the format or nature of these online 
communities, the underlying commonality that is pervasive among these rampant computer-mediated communities 
is the social dimension of communication. Thus, the complexity of communication in online communities caused 
by the void of face-to-face interactions and text-only discourse provides a fertile ground for the study of human 
interactions in these evolving virtual communities of practice. Much recent study on virtual communities has been 
devoted to topics on gender-related issues, online participants' communication styles, potentials and challenges of 
online communities within the realm of education (Ferry, Kiggins, Hoban, & Lockyer, 2000; Kreijns, Kirschner, & 
Jochems, 2002; Lapadat, 2002; Rogers, 2000; Soukup, 1999). Moreover, the impacts of online communities on the 
society as a whole, and the social interaction and communication of related issues are reported in the literature 
(Agres, Igbaria, & Edberg, 1998; Cummings, Butler, & Kraut, 2002). 

In studies of both face-to-face and online conversation, the social and communicative significance of 
seemingly non-sensical conversation has been well documented (Preece, 2000). However, in online discussions in 
which teachers engage in professional conversation, "off-topic" or "whimsical" dialogue is often viewed as off task 
or time wasting. This paper examines the social aspect of a particular synchronous Book Talk among elementary 
reading teachers and describes preliminary findings regarding positive effects of seemingly nonsensical 
conversations in this environment might, in fact, support the development of particular aspects of a community of 
practice (Wenger, 1999). 



Conceptual framework of the study 

Much literature has addressed the different social aspects of community of practice, particularly in 
association with the burgeoning virtual communities in recent years. Moreover, studies on analyzing the actual 
dialogues or discourse patterns of virtual communities are diverse in scope and in theoretical framework. For 
example, Muramatsu and Ackerman (1998) studied the social management and arrangement of participants in a 
combat MUD community. From observations of the discourse among members of the game participants, the 
researchers found that meaning making resulted largely from collaboration and conflicts. Their dialogue analysis 
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also identified significant differences between the social interactions of members depended upon the purpose of the 
virtual environments— whether for entertainment and work for example. In a longitudinal study by Schoberth, 
Preece, and Heinzl (2002), actual email postings were analyzed for an understanding of the social behaviors of the 
online communities. Their quantitative analysis of three years' email messages from a German financial service 
provider revealed high correlation between information load, communication strategies and communication 
activities. Herring (1999), in her study of the structural and behavioral patterns of computer mediated textual 
communication, sought explanations for the complex incoherent nature of synchronous and asynchronous interactive 
exchanges. By analyzing samples of online dialogues. Herring (1999) found that the prevalent problems of topics 
decay, separated response sequence, exchange gaps and exchange overlapping in today's technologically determined 
online environments often create confusion and incoherence in their textual dialogues. However, her study indicated 
that online participants frequently sought compensatory strategies to adapt. Furthermore, many online users extend 
their creativity in the textual form of expressions by playing with words and manipulating textual expressions. 
Although the aforementioned literature concerns social interaction and communication, these studies do not explore 
how the complexity and multi-dimensions of social exchanges in online conversation relate to the development of a 
community of practice. Toward this end, the authors adopted Wenger's (1999) definitions of community of practice 
as the conceptual framework to examine the dialogues of a synchronous Book Talk among reading teachers. 

According to Wenger (1999), a community of practice is one in which members engage in the same types 
of work and thereby experience a continual process of unfolding, integrating, negotiating, and development of 
related knowledge, expertise and support. The interwoven relationships of members in these communities of 
practice encompass three essential dimensions. They are mutual engagement, joint enterprise, and shared repertoire. 
Each of these dimensions has unique characteristics of interaction that maintain the coherence of the communities. 
For instance, mutual engagement entails the process of evolving forms of relationship and community maintenance; 
joint enterprise encompasses the process of understanding and turning their enterprise; and shared repertoire embeds 
the process of developing their styles, artifacts and discourses (see Figure 1). 




Figure 1. Dimensions of practice as the property of a community (Wenger, 1999, p.73). 

As much online conversation is text-based, not surprisingly, it differs from face-to-face 
communication. Burnett's (2000) has developed a typology of social exchanges in virtual communities. According 
to Burnett (2000), both of these categories can be further divided into non-interactive or interactive behaviors of 
online participants. Within the interactive category, online talk and behavior can be either hostile or collaboratively 
interactive. In turn, the collaborative interaction category is further subdivided into behavior not specifically 
oriented toward information and those that are. The initial focus of this preliminary investigation of the 
conversation in a synchronized teacher professional development forum (chat) was on the information-oriented 
behavior in the Informative Interaction category. However, the authors began to question their tendencies to dismiss 
the joking or otherwise seemingly the non-informative interactions as irrelevant. 
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Methods 



The Forum Description: Teachers.Net Book Talk 

This qualitative study focuses on the archived transcript of a synchronous Book Talk session in 
Teachers.Net services, which is a division of Teachers Web. The motto of teacher-oriented website is "improving 
education through technology and collaborative support. ..together we share an incredible potential to change the 
face of education" (Bott, 2002, f 1). The site is supported by educational organizations and "commercial enterprises 
seeking technical and promotional solutions to researchers of cutting edge theory and applications" (Bott, 2002, f 2). 
AIG VALIC, an organization that offers financial services and products, is the major sponsor; others include Azusa 
Pacific University, Apples 'N' Acorns, LASIK for Educators, Meet the Masters NRTA, ReadingLady.com, and VIE. 
Aside from educational advertising by the sponsors, there are numerous instruction-related chat rooms, chat boards, 
live meetings, and topical meeting archives. Among the diverse chat forums in Teachers.Net, the archived chat 
meeting appeared to be the most appropriate for the observation of online community behavior. These archives are 
transcripts of live events in areas such as phonemic awareness discussions, or grade teachers' chat. Generally, live 
chat or discussion sessions take place at 9:00 p.m. Eastern Time in the Meeting Room. 

In order to entice advertisers, Teachers.Net reported on their general demographics obtained during a 
seven-day data collection from 1200 voluntary participants taken in the fall of 1998. According to the note to 
potential advertisers, this Teachers.Net survey claimed to have 91.87% of teacher participants who made up 90% of 
the female surveyed population. Furthermore, among the 95% of college educated participants, 44.4 % of them 
received advanced degrees and the average age of the surveyed participants was 40.6years (see Table 1). 



Categories 


Unit Breakdown 


Surveyed Results 


Age of Participants 


Average age 


40.6 years 


Curriculum Taught 


Early Childhood 


16.5 % 




Elementary 


54.2 % 




Middle/HS 


21.9% 




Other 


7.4% 


Education (Highest 
grade achieved) 


HS 


0.25 % 




Some College 


2.68 % 




Assoc. 


1.76% 




Bachelor 


51.26% 




Masters 


41.36% 




Doctorate 


2.68 % 


Ethnicity 


Caucasian 


93.32 % 




AA 


2.23 % 




Hispanic 


1.71 % 




Multi 


1.63 % 




Other 


1.11 % 


Table 1. Teachers.Net survey results - fall 1998 


Categories 


Unit Breakdown 


Surveyed Results 


Gender 


Female 


90.7 % 




Male 


9.3 % 


Marital Status 


Married 


73.8 % 




Single 


26.2 % 


Occupation 


Teacher 


91.87 % 




Ed. Student 


3.6% 




Admin. 


1.97% 




Other 


2.56 % 


School Type 


Public 


85.2 % 




Private 


12.0 % 




Other 


2.8 % 
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Student Age 


Average 


8.8 years 




5-9 


52.0 % 




10-14 


26.1 % 




15-18 


7.3 % 


Teaching 


Other 


14.6 % 


Experience 


Average 


13.66 years 



Table 1. Teachers. Net survey results - fall 1998 



Like all the other free Teachers.Net chats and meetings, the chosen archived session. Book Talk, met as a 
scheduled real-time event. Starting its first session on April 17, 1998 with seven teacher participants. Book Talk has 
been meeting irregularly until March 2002 and continued weekly until the beginning of 2003. 

Participants 

Based on the content of the Book Talk transcript, participants were from various parts of the United States 
such as California, Michigan, Tennessee, Oregon, Illinois, and Virginia, and Canada. Gender, race, and ethnicity 
information was not available explicitly. However, gender was inferred by participant names, although we had no 
way to confirm our inferences. Participants also did not specify grade level they taught, although again, it appeared 
from the topics that several were elementary teachers. Participants were given pseudonyms in the Excerpts of chat 
logs that appear in this report. 



Procedures 

Access 

The interest of the authors to explore the engagement of teachers in professional online discussions led to 
the search on teachers' discussion boards, chartrooms, and synchronous meetings. However, not many of the teacher 
related websites provided easy access to the information of online discussions without being a member. The 
selection of Teachers.Net among other similar teacher's online communities was largely due to the fact that 
accessibility regulation to the events and archived transcripts were readily available. For the purpose of this study, 
the authors chose Book Talk among other discussion sessions in the archive for the simple reason that the dialogues 
appeared to be lively and amusing. The authors contacted the online administrator of Teachers.Net to seek 
permission to use the transcriptions of Book Talk chat archives. However, we did not receive a reply. A 
consultation with our Institutional Review Board (IRB) indicated the exemption status of the current study as the 
logs of Book Talk chats are a publicly available resource and moreover had been posted on Teachers.Net before this 
study began. 

Sample 

Sampling in conversation analysis (CA) precedes from a different starting point, with different 
epistemological assumptions than that of sampling in quantitative frameworks (Mazur, 2004). Drawing on the 
insights from naturalistic observation and biology, conversation analysis justifies the use of 'specimens' of 
conversation. Specimens (in biological study, for example) represent the phenomena or organism under study that 
are directly observable in the particular individual. While such specimen sampling may have criteria, there is no 
need to sample statistically. For example, a biologist examining a species of tadpoles would simply choose 
individuals from that species. Perhaps, by observing the species in some natural setting, the biologist would not 
choose one that seemed highly unusual, but there would be no need for a statistical sample of tadpoles since it is the 
category of specie, the specimen, that is of interest. Similarly, a CA study might focus on any category of talk-in- 
interaction such as repairs and then simply select any specimen of conversation generated in a naturalistic setting. 

As Hutchby (2001, p. 51) has claimed, "The logic of CA, however, in terms of data selection suggests that any [his 
emphasis] specimen is a 'good' one, that is, worthy of intense and detailed examination (as quoted in Mazur, 2004)" 

The Book Talk session took place on November 1, 2002 with 16 participants and 1025 exchanges. 

However, the observed portion has 50 exchanges among eight regular participants. The major themes of the selected 
discussion session centered on an upcoming conference, personal features such as hair color and age, school 
substitute issues, Halloween party, number of school days, phonics, readings, and issues related to teaching and class 
schedules. For analysis purposes, the archived chat logs were copied and formatted, which yielded 46 pages of 
transcripts. For ease of reference, the dialogue of the whole discussion session was numbered, which resulted in 
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1025 lines of conversation. Lastly, the names of participants were italicized and were given pseudonyms. From the 
formatted conversation, the researchers chose a specimen comprised of lines 71-121, that was termed the “Hair 
Talk” to enable generalization of observations on the talk-in-interaction of participants. Expecting a discussion of 
substance on children's book and reading in general, the researchers were initially disappointed then intrigued by the 
somewhat silly repartee regarding hair color and style that ensued. What meaning could such nonsense have in this 
self-initiated interactive context? 

Gunawardena, Lowe and Anderson (1997) have cautioned against the sole use of conversation analyses to 
characterize the interactions in online forums. Their concern is that the focus on the details of talk-in-interaction 
may result in the loss of the so-called bigger picture, the meanings, of the conversation. They justify their position 
as follows: 

The totality of interconnected and mutually-responsive messages which make up the conference, and 
perhaps more: 'interaction' is the entire gestalt formed by the on-line communication among the 
participants. The process observed in the debate is akin to thinking on 'distributed cognitions' (Salomon, 
1993, p. 256) where he states individual and distributed cognitions interact over time, affecting each other 
and developing from each other, (p. 407) 



Analysis 

Using the frameworks from both traditional conversation analysis (ten Have, 2000) and those that focus on 
online conversation (Herring, 1999), the Book Talk online discussion was analyzed. Turn taking, sequencing and 
content of conversation were examined for normative patterns in the conversation. These patterns were then related 
to Burnett's (2000) typology for exchanges in a virtual community and to the concepts from Wenger's (1999) notion 
of a community of practice, specifically, mutual engagement, joint enterprise and the development of a shared 
repertoire. 

Analyzing the turn taking organization in a conversation is of foremost important, and is "one of the core 
ideas of the CA enterprise" (ten Have, 2000, p. 111). The basic idea of turn taking is the procedure of conversation 
that carries from one speaker to the next. During a normative conversation, the speaker selection, the tone of 
speaking, and the gaps in between responses are just some of the phenomena of turn-taking organization. Since 
online written dialogues do not exhibit the tone of speaking as in face-to-face communication, expressions such as 
exclamation and smiling symbols are often used instead. With speaker selection, the three prevalent ways that 
speakers exhibit their turn taking during a conversation are other-selection, self-selection, and continuation (ten 
Have, 2000). Each turn taking option illustrates the participants' actions within the context of the conversation. 

Findings 

Turn-Taking 

Continual and self-directed turn taking was evident throughout the selected conversation specimen as 
demonstrated by the greetings of Terri, Todd, rita, amy, and kevin (see Excerpt 1, 71-75) to Annie. No one in 
particular suggests them to welcome Annie. In response, Annie does not hesitate in returning the greetings. Line 77 
(see Excerpt 1) typifies an other-directed organization. Amy tries to find out the reason for Todd's earlier statement 
about his hair, which arouses her interest enough to direct the question to Todd. 

The significance or likewise the insignificance of certain participants in the community often surfaces 
through turn taking analysis. Erom the same sample as above (see Excerpt 1, 71-76), all conversation come to a halt 
when Annie logged on to the session. The members welcome her with all kinds of expressive symbols (emoticons) 
to show their excitement in seeing her. On the other hand, no one acknowledges ted's presence when he starts to join 
the conversation (see Excerpt 2, 93). His statement regarding rita's hair is ignored. It is possible that he has been 
observing in the room for some time or perhaps he has just logged on and there is a delay in the posting his message. 
Either way, he does not seem to care about any acknowledgment of his presence nor is he concerned with invoking 
any further actions in announcing his presence. 



Excerpt 1 

7 1 . Terri 


(((Annie))) 


72. Todd 


Hi Annie!!!!!!!! 


73. rita2a 


(((Annie))) 


74. amy 


Hi Annie! 


75. kevin 


Hey Annie!!! 


76. AnnieL 


Hi Everyone! 
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77. amy 



Todd, why will you have small animals in your hair? 



Sequence of Conversation 

Very often, sequence interactional organization goes hand in hand with turn taking organization. Both of 
them are the backbone and core idea of conversation analysis. The basic concept of sequence organization is the 
implied response that is to follow a prior dialogue. For the reason of relevance to online conversation, the authors 
will include only responsive actions and presequence actions in the discussion. Response actions are frequently 
displayed in various forms of interactional conversation. Responses such as agreement, disagreement, acceptance, 
rejection, or correction (Goodwin & Heritage, 1990) are often associated with the promptness in responding to the 
speaker who initiates the statement, question or comment. Presequence actions is characterized by "preface 
requests, jokes and stories. ..news announcement... and invitations" (Goodwin & Heritage, 1990, p. 297). They often 
cause a change of action or response on the part of the hearer when faced with conflict. 

In "The Hair Talk", Todd was the first to disclose information about his curly blonde hair (see Excerpt 2, 

77, 78, 79, 81, 83, and 84) when amy asked about the "small animals" in his hair (see Excerpt 2, 77). By having an 
open ended sentence in his first reply (see Excerpt 2, 78), Todd seems reluctant in disclosing any information about 
his hair by avoiding to answer amy's question. However, he changes his mind by finishing with his reply on the 
next two entries of response (see Excerpt 2, 78, 81). Erom then on, Todd wants to find out the hair color of others 
by poking for information (see Excerpt 2, 95, 97, 103, and 108). Posting his question to kevin (see Excerpt 2, 97), 
Todd generates a seemingly reluctant responsive action (see Excerpt 2, 101) from kevin. Instead of answering the 
question that Todd posts (see Excerpt 2, 97), kevin raises a question in return, (see Excerpt 2, 101) which leads Todd 
to ponder on giving the response (see Excerpt 2, 102). Kevin's question (see Excerpt 2, 101) also signifies a pre- 
announcement that induces Todd to guess rather than to give him the answer. In the same token, the question that 
Todd directs to ted regarding his hair color (see Excerpt 2, 95) causes a similar reluctance in response from ted (see 
Excerpt 2, 99). The manipulation of responses by kevin and ted enables them to tease Todd. Todd's frustration in 
getting no reply from others is evident in his response in line 104. 



Excerpt 2 

77. amy 

78. Todd 

79. Todd 

80. rita2a 

81. Todd 

82. kevin 

83. Todd 

84. Todd 

85. kevin 

86. rita2a 

87. kevin 

88. rita2a 

89. Todd 

90. AnnieL 

91. Todd 

92. rita2a 

93. ted 

94. kevin 

95. Todd 

96. AnnieL 

97. Todd 

98. Terri 

99. ted 

100. rita2a 

101. kevin 



Todd, why will you have small animals in your hair? 

Because I have so much of it 

Even the guy who does my hair jokes about it 
Todd, LOL, I have straight used-to-be-blonde hair. 

It's naturally curly, which I now view as a blessing. ..but it is a chore to take care of 

todd you are not a blonde you are defmitly a redhead I can tell from here 

Mine is blonde. .with a little help. I have a lot of gray for someone who is 35, but well 

covered 

kevin. ..I do have some red in there too LOL 
Rita, you are a brunette 

Todd, my Lit Coach (who I'm going with) isn't going to Keene's presentation— so I'll be 

alone there. I'll definitely look for you. 

you guys are ruining my ESIA member schema 

kevin, ROLL 

I told my fellow teachers I'm dropping my bag at the hotel and heading over. They can 
check me in 

The presentation is next week? 

Yes, Annie. I get to meet Rita 
yup Annie 

Rita is NOT a brunette 
I know that now. I'm trying to adjust 
ted, are you a brunette? 

No, Rita is blue, 
kevin. ..and you? 

Terri's gray — just letting it do its thing. 

natch 

ROELOL 

What do you think Todd? 



247 




102. Todd Hmmmmmmmmmm, ted, blond? 

103. Todd Ummm, well, uh, blonde? kevin? 

104. Todd Thanks everyone for leaving me out here all alone. LOL 

Content of Conversation 

The content of this non-sensical “Hair Talk” conversation reflects a distinctive tone and style. In excerpt 3 
below, there is evidence that these participants really know their fellow Book-Talkers in ways that suggest a sense of 
belonging and true insights into identity. Participants appear to be very comfortable with each other and they seem 
to know each other well enough as to tease (perhaps even bait) each other. 



Excerpt 3 

71. Terri 

72. Todd 

73. ritala 

74. amy 

75. kevin 

76. AnnieL 


(((Annie))) 

Hi Annie!!!!!!!! 
(((Annie))) 

Hi Annie ! 

Hey Annie!!! 

Hi Everyone! 


^ Color the Atmosphere 


86. rita2a 


Todd, my Lit Coach (who I'm going with) isn't going to Keene's 
presentation— so I'll be alone there. I'll definitely look for you. 


91. Todd 


Yes, Annie. I get to meet Rita 


117. rita2a Todd,I'm a lot older than you, but I'll probably leave the 

cane at home. 

118. kevin And I have thin thighs, long legs and buns of steel too 



J 



Members 
appear to 
know each 
other well 



Discussion 

These seemingly non-sensical online interactions of "The Hair Talk" conversation derive purpose in their 
potential to support dimensions of activities inherent in communities of practice. Specifically, within the dimension 
of mutual engagement, members of the community discover ways to interact with each other, build relationships, 
establish identifies, find out others' identities and explore their strengths and weaknesses. The self-directed and 
continual greetings when someone logs in reveal the willingness of Book Talk participants to develop and maintain 
engaging, welcoming mutual relationships (see Excerpt 1, 71-76). Their greetings also signify their 
acknowledgment of others' identities. For instance, through other-directed organization, amy, one of the eight 
participants, learns how to develop a repertoire (styles, actions, artifacts) with Todd. Her initiation in questioning 
about Todd's hair is intentional and meaningful (see Excerpt 1, 77). 

Members in a community tend to share a common goal (joint enterprise). They learn through the process of 
understanding and tuning their enterprise. In the process of working together to solve disagreements and conflicts, 
members develop mutual accountability, and align their engagement from negotiations. From the specimen, it is 
evident that Todd's frustration indicates his struggles in defining the enterprise and his coping with the conflict. His 
effort in resolving the confusion at hand indicates his accountability to the other participants. The way that ted and 
kevin lead Todd into frustration demonstrates their understanding of their own community (albeit put to somewhat 
negative ends), the members, and the boundaries within the community (see Excerpt 2, 93-104). 

According to Rogers (2000), "meaning is negotiated in a community through its shared repertoire. This 
repertoire refers to the fact that there is a pool of resources that members not only share but also contribute to and 
therefore renew" (p. 388). New ideas are generally created through the shared repertoire. For instance, despite the 
fact that this particular online chat is a book discussion session, the participants have no reservation to renegotiate 
the meaning or purpose of the discussion session (see Excerpt 2, 87-94). They develop their own style in relating to 
and in learning about each other. These interactions also indicate a strong sense of belonging. They also learn to 
color the atmosphere of the chat environment by showing excitement in welcoming others (see Excerpt 3, 71-76), 
and being playful. 
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Conclusion 

Initially, the selected Book Talk conversation appeared to be trivial nonsense, especially considering the 
fact the participants were reading teachers in a book discussion session. The commentary on each other’s hair color 
do not have any bearings on the rest of the conversation related to upcoming conferences, specific books and 
upcoming holidays at school. In fact, the conversation seemed somewhat immature. On the other hand, this 
mundane conversation, which is prevalent in many ordinary social conversations and interactions, may possibly 
have significant implications. 

The selected specimen of exchanges were primarily non-informative (Burnett, 2000). Within the 'non- 
informative' frame, the analysis showed that this kind of seemingly irrelevant online interaction supported a) mutual 
engagement, b) joint enterprise, and c) development of a shared repertoires - three activities related to the 
maintenance of coherence in a community of practice (Wenger, 1999). 

The relevance of seemingly off topic or tangential utterances should be no surprise to those who are 
familiar with conversation analysis of talk in face-to-face contexts. However, systematic analysis of how these types 
of interactions play out in on-line chats has been lacking. These reading teachers log in week after week to 
Teacher.Net seeking out other teachers. For them it appears that there is potentially much sense in the nonsense of 
seemingly non-informative online interactions as they may relate to the development of a community of practice. In 
fact, the insights reported here about these non-informative interactions may help to explain the persistence of these 
voluntary, online professional communities. Further analysis of the details on online talk-in-interaction my reveal 
additional information regarding how online talk relates to the building, maintenance or erosion of communities of 
practice. 
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Using a Wiki Learning Environment in a Secondary Education Classroom 



Cory Little 
Utah State University 

"Not all advances in instructional technology come about through the development of new hardware or 
software - some emerge from the creative applications of existing technology." (Wiley & Edwards, n.d., p. 2, 
emphasis in original) 



What’s Wrong with What We’ve Got? 

The middle school bell rings and 32 students in second period history stand up and file out the door on their 
way to English or math or band. Over the next five minutes 34 third period students trickle in — some to find their 
seats and pull out reading books, others to chat with a friend, and others only to drop their books on a desk and rush 
back out for a couple of minutes of socializing and note exchanging in the hallway. As the tardy bell rings, all 
students rush for their assigned spots and, as the last bell tone dies, the last student’s bottom hits the last open chair. 
Pavlov would be impressed! 

What happens between the starting and ending bells of a class period may vary widely from day to day, but 
in the broadest sense it is hoped and assumed that learning is occurring. It is also a pretty safe bet that instruction is 
expected to play a major role in that learning process — particularly at the middle school level. What many people, 
including teachers, seem to forget is that the practice of instruction and the practice of learning are not simply two 
sides of the same process. This is because the teacher's goals of instruction may or may not be the same as the 
students' goals of learning — either for a single student or for an entire class. It may also be the case that the methods 
that the teacher uses for instruction are incompatible with the student's mode of learning. 

Perhaps the term “education” is general enough to cover both processes. Education may refer to the 
teaching by a single instructor or by the whole school system. Education may also be used to describe what a learner 
“gets” in a classroom — the results of their participation, or the process by which those new abilities and 
understandings are acquired. Ask, then, of the goal of education at the secondary level and you are more likely to 
find general agreement among teachers: Students will: (1) learn the subject matter for core courses that prepare them 
for college and/or productive careers, (2) become self-regulated life-long learners, and (3) develop better moral 
character or citizenship. So, how are we doing on those three fronts? In most seventh-grade classes we are doing 
very poorly! The ideal student is one who sits quietly and does each assignment without questioning the teacher, the 
assignment, or the textbook. We test thoroughly to see if the students retained the subject matter at least until test 
day; we encourage self-regulation unless they get 15 minutes behind; we reduce “life-long” to the end of the current 
grading period; and we convince them that citizenship refers only to the number of times they talked without raising 
their hands or strayed from their assigned seats. 

Is There Any Other Way? 

Most of the burden for initiating, maintaining, and evaluating learning falls upon the teacher. Because of 
that, the emphasis is on instruction rather than on learning. A number of educational theories that have been 
developed over the past few decades, loosely combined under the label “constructivism”, seem to pull the focus of 
education from teaching to learning. Even an understanding of what learning involves changes with a contructivist 
approach — from reproduction of information to the notion that information must be transformed by the learner for 
meaningful learning to occur (Cullen, n.d.). Constructivist learning requires that students, preferably groups of 
students, be able to work with information, cognitively and socially, to make the information meaningful for the 
individual learners. As Cullen (n.d.) points out, constructivist learning is a negotiated process. Negotiation occurs 
between the teacher and student regarding learning activities and the management of the learning environment. 
Negotiation may also occur regarding the selection and use of sources of information. Negotiation is also critical 
between individuals in the social learning environment. 

Negotiation implies a shared understanding of the division of responsibility and authority or control 
between the teacher and the students in the learning environment. Despite educators’ goals of developing self- 
regulated life-long learners, they are very reluctant to surrender much control to the students. When they do, it is for 
a very short time. A lot has been written about giving students control over their learning - some authors emphasize 
the benefits while others focus on the negative consequences. One point on which most authors tend to agree. 
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however, is that students and teachers generally lack the skills and the experience necessary to negotiate effective 
learner control. 

There is yet another point of negotiation that I think is the most critical for learning — the interaction 
between the learner and the source(s) of information. Although it may be a stretch of the definition of negotiation, I 
believe that it effectively describes the way in which a student interacts with a lecture, a passage in a textbook, a 
discussion with a neighbor, or any other source of information. In a one-on-one conversation with a teacher or 
another student the learner has considerable control. Even in a class-wide lecture situation there are opportunities for 
students to interrupt the flow of information to ask for restatement, clarification, or extension of ideas, though such 
interruption is sometimes challenging to accommodate with large classes in the mass (public) education system. 
Ideally, students can negotiate meaning with others until all parties are completely satisfied, but it is not so easy to 
negotiate with a textbook. 



The Holy Textbook 

The textbook has been the center of attention in most secondary social studies classrooms for several 
generations. It brings together in a concise form much of the information that is relevant to a given topic, 
accompanied by photos and maps and other learning aids. It organizes the information in a logical way, and it makes 
the exact same information available to every student in the class and to the teacher. 

Now if students would just read it. So teachers give worksheets and assign questions within the chapter to 
be answered to make sure they do read it. But they still don’t. (After first making sure that no one around them has 
the answer to give them), students scan the text for one of the key words in the question. Likely as not, the teacher 
will get the next seven words, directly from the text, as the answer — whether or not they even make sense. Students 
will have forgotten the question and the answer before they even move on to the next one, and as for understanding 
it or thinking about it — not a chance. So we assign them to take notes or to write an essay about the information in 
the chapter. What we get from most students is a brief synopsis of the chapter (a few lines verbatim from the text)— 
long enough to claim that they worked very hard and yet sloppy enough so that the teacher doesn’t want to read it 
carefully to see that they didn’t. 

How do you get kids to actually think about what they read, or see in a video, or hear in a class discussion? 
How do you get students to become critical consumers of information, who then actively and purposefully transform 
that information into meaningful learning in the constructivist sense? I think a major part of the problem is the 
textbook and the way it is used in the classroom. Siegel and Sousa (1994) complain that the traditional textbook, 
“with its emphasis on chapters and units, dictates the flow of ideas in the classroom. It does not easily allow students 
and teachers to depart from the prescribed path.” (p. 49) The path the text takes has probably been prescribed by a 
group of professionals who are experts in the subject matter, in pedagogy, and in editing. It largely controls the 
sequencing, pacing, and depth as well as the content of what is to be learned. Teachers appreciate the fact that the 
difficult task of locating, selecting, and sequencing information is done for them — leaving them the monotonous but 
easy task of correcting worksheets and recording scores. But students are led to think that the chapter contains all 
relevant information, organized in the only logical way. 

It’s the organization of knowledge that really concerns me. Apparently it is of great concern to nearly all 
learning theorists as well. Many years of serious thought and research have been spent trying to understand how the 
human mind encodes and retrieves information and how we give meaning to that information. Scores of cognitivist 
and constructivist theories attempt to describe how the mind works and to prescribe instructional approaches based 
on those understandings. Siegel and Sousa’s (1994) sharpest criticism of traditional textbooks is that teachers and 
students are forced down the prescribed path in a linear fashion that “neither reflects nor encourages natural human 
learning.” (p. 49) Although the expert path may be appropriate when learning some skills, such as parallel parking, it 
is clear that this approach can be very limiting in a subject like history, where links to a wide variety of concepts and 
facts, both within and without the history discipline, are important towards increasing comprehensive understanding. 
Even the experts cannot prescribe a single path that is best for multiple individual students. 

How can students get the information that they need about a subject, with enough structure to keep it 
coherent, but enough flexibility so that it more accurately follows the way they individually process information? 
How can they practice metacognitive skills that would help them to monitor and adjust and improve their own 
processing of information according to their unique, individual abilities and interests? Allow students to assume 
more control over the information that they encounter, particularly over the way it is organized! 
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Creating Multiple Texts from One 

Computer and Internet technologies make it possible to do many things with information that we cannot do 
with printed books. Although there may be some drawbacks to the rise of electronic writing and reading, such as a 
decline in the concern for spelling skills, the benefits far outweigh those drawbacks. One innovation in particular, 
hyperlinks, allows the reader to effortlessly and immediately link to additional information and back again at will. 
Hypertext, a term coined by Theodor Nelson in the 1960’s (Landow, 1992), attaches links to additional information 
to specific words and phrases in the online text. Even very young children know as they surf the Internet that 
clicking a word that appears in a different color, usually blue, and especially if it is underlined, will take them to 
additional, related information. 

Twenty years ago a number of children’s books appeared that allowed the reader to determine the way the 
story unfolded based on decisions that they made. A second reading of the book, with a few different decisions by 
the reader, would likely produce a different ending to the story. In much the same way, hypertext allows the reader 
to determine, to some extent, the way in which a document will be read, based on the links they choose to follow. 
Landow (1992) wrote that “hypertext blurs the boundaries between reader and writer.” (p. 5) This occurs because 
the reader assumes some control over how the document will be read and continually changes the focus by selecting 
a particular path of hyperlinks. Spiro and Jehng (1990) describe the different paths that might be taken as producing 
“essentially multiple texts for the same topic.” (p. 166, emphasis in original) Lehrer (1993) describes the paths as 
“multiple voices” (p. 201) and I think that describes the effect of hypertexts well because it suggests that multiple 
readers will have different experiences as they travel through the larger body of work. The key is that the reader has 
been given some control. 

So, putting a hypertext textbook online would solve the textbook issue, right? It would make the textbook 
less constraining and it might also make it more accessible. However, it makes the management of learning activities 
and evaluation of learning more challenging for the teacher if each student is now reading their own unique 
“version” of the text. And it really doesn’t address the more serious problem — that students don’t really read the 
textbook anyway. They need to have a reason to carefully, critically, and repeatedly read the text — they need to 
write it! 



Make Me Want to Write 

Few students like to write, and almost no students would be excited about the prospect of reviewing and 
revising their work to the degree that would be necessary to produce a basic text on any subject. From the hundreds 
of excuses students may have for not writing well, only a handful can legitimately be resolved by direct instruction. 
The majority of excuses boil down to a lack of motivation on the part of the student. Vincent (1993) describes three 
conditions that must be met before students will become totally involved in major writing projects; "(1) they must be 
turned on by the project; (2) it must include a broad, well-defined audience; and (3) the project must be in some 
sense 'published' to convince them of the importance of proofing or editing." (p. 58) Although the teacher may 
employ some strategies to encourage motivated participation in such a project, I believe the most effective strategy 
is to construct a learning environment that features three different motivating factors: technology, increased learner 
control, and positive social reinforcement. 

Finally! - The Wiki Environment 

The Wiki is an online environment that incorporates many of the ideas that have previously been 
mentioned. It is a workspace where students can create and modify hypertext documents, and it becomes a 
showplace for their evolving masterpieces. Developed several years ago by Ward Cunningham and his colleagues at 
Georgia Tech (Leuf, 2001), the wiki is far more than a web site where students post their work. The wiki server 
allows students — actually anyone who views the site — to add to or to edit existing content and to manage text pages 
with the click of a button and without any technology skills beyond basic keyboarding. 

As the wiki learning environment is described throughout the next few pages it may be helpful to explore 
an example of a very successful wiki site on the Internet. One site that seems to work well because of its large, 
active community is an attempt to create an online encyclopedia based on the distributed expertise of the users. This 
is found at www.wikipedia.org . The wiki that I used for my classes was hosted on a server belonging to Dr. David 
Wiley at Utah State University. Students access the wiki environment using an Internet browser. Most wikis are very 
simple in terms of their layout so that they can be accessed by nearly any computer and any browser. Within the 
wiki the student may choose to start a new topic by naming a page with a "wiki-word" title. A "wiki-word" is 
formed by putting two capitalized words together without a space, such as WealthyNations. Unless this topic is later 
renamed, it will always be available in the wiki under that name. If a student wants to link to another topic that they. 
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or another student, has written, the student simply types in the wiki-word name of the desired topic and the link is 
automatically created. The wiki word is also shown in blue to indicate that the link has been created. If the student 
creates a link to a page that does not yet exist, the link will still be shown in blue, however it will be followed by a 
question mark to indicate that the page does not yet exist, or that it does not yet have any content. If a student clicks 
on a link with a question mark, the blank topic will still be shown and the student can then add content to that topic. 
When their work is saved, the new topic is available online and the question mark after the link will be removed. 

The organization of topics within the wiki is controlled only by the strategies students use to link to them. 
Some topics flow naturally from one to another as shown in Figure 1 . Others could be linked more as a web as 
shown in Figure 2. Ideally, topics would be limited to individual ideas so that there is more flexibility in linking 
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them together. As the number of topics on a particular subject increases, the organizational possibilities also 
increase— this is an important feature of the hypertext learning environment. 

As students read existing topics, they also have the opportunity to edit those topics. Anyone can edit any 
topic in the wiki, regardless of who created it initially. When a student clicks an edit link at the end of the topic, that 
topic is locked to other users so that only one person at a time can make changes. Changes are made to the topic as 
they would be in any basic word processor. Text can be added, deleted, or replaced. Links to other topics can also be 
changed or added. When the changes are done, the student clicks a save link and the server immediately updates the 
topic. If the student returns to the Web browser, they can see the new version of the wiki, though they may need to 
refresh the browser first. 

The last saved version of each topic is displayed in the wiki. Flowever, all previously saved versions are 
also readily available. A previous version can be restored simply by selecting it and saving the topic again. This may 
be valuable in an educational situation where topics become accidentally or intentionally corrupted. Because anyone 
can change any topic at will, it is possible for students to damage other students' topics. However, because unwanted 
changes can be discarded with a couple of mouse clicks, damaged wiki topics are not a serious threat and students 
who intentionally cause damage soon learn that their efforts are not time/cost efficient. 

Most wiki environments are set up so that there are at least two separate publication threads going on at the 
same time. One thread is the main web of topics that the students write for all to see. Behind the scenes, however, is 
a second discussion thread that students use to carry on a kind of meta-conversation about the topics they are 
publishing. The discussion side of a topic may include extended explanations for the topic's contents, justification 
for changes they have made, or questions that are yet to be addressed. 

In addition, there may be meta-pages for the entire site. These may include guidelines for site development 
that the community of learners might follow that influence organization of topics or the directions the content 
development will go. This area would also contain a help page with a users' guide for working within the site. This 
page may also include a place where students (and others) can post questions to be answered by any other user. If 
the wiki gets very complex and the community is very active, it would be necessary to assign one or more students 
to monitor the meta pages and maintain some degree of organization. 

As the students become familiar with the affordances and the flexibility of the wiki environment, it will 
begin to develop a particular style. It may take some positive experience with the environment, but students will 
eventually become very familiar with how the wiki works and a core group of students will emerge who show great 
interest in, and responsibility for, the wiki. Although it is difficult to isolate independent factors that make the wiki a 
motivating and productive learning environment, I think it is important to consider the three aspects that were 
previously mentioned: technology, social reinforcement, and learner control. 
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Technology 

As a whole, the research findings are inconclusive regarding the use of technology in schools simply 
because there are so many different variables and combinations of variables that are included in the many studies. 
The same is true for the kind of technology that the wiki laming environment represents. However, two meta- 
analyses on hypertext and hypermedia (Chen & Rada, 1996; Liao, 1998) acknowledged positive overall effect sizes 
when using such tools. These reviews of studies indicated improvements not only for learning outcomes, but also for 
social interaction. Such tools or environments encourage students who typically do not get involved in traditional 
classroom discussions or in face-to-face exchanges to communicate with others (SRI International, n.d.). Because 
working in the wiki is so easy, it encourages participation by all of the students. They see the results of their efforts 
immediately, which further increases their motivation to continue. 

Two concepts associated with the constructivist approach to education are highlighted by the technology of 
the wiki. The first is the concept of "affordances", a term coined by Gibson (Driscoll, 2000) to describe the impact 
of the environment on an organism's behavior. Gibson argued (Driscoll, 2000) that an organism's behavior, in this 
case that of a student, was controlled by its perception of the affordances of the environment. A student who thinks 
they will learn something new from a movie about tigers is more likely to pay close attention to the movie and is far 
more likely to learn than a student who perceives the movie as a waste of time. Riel (1989) describes the use of 
computers in the classroom as a "recontextualization of learning" (p. 188). I think that describes what happens when 
students encounter the same information in the wiki environment that they might otherwise have seen in a textbook 
or heard in a lecture. The new environment gives the same information new life for the students. Student interest is 
focused on both content and on the tool used to present it. The wiki, despite its simplicity of use, can be used to 
create very intricate hypertext webs. As students perceive the possibilities that the wiki, as a tool, can provide, they 
will be further interested in using the tool and in exploring those possibilities. 

One of the more practical affordances of the computer and of the wiki is the ease with which changes can 
be made. Although most writers make minor changes as they write, students revise their work more frequently when 
they are using a computer (Soven, 1999). This revision includes not only simple editing, but holistic revision of their 
work. The computer screen becomes a focus for students' attention. Unlike a textbook, the screen is constantly 
changing as the student's focus shifts and as they update the information there. The screen ajfords the student the 
ability to see the results of their construction of knowledge (Tourney-Purta, 1996). Because it is so easy to make 
changes, and because those improvements can be immediately seen on the published wiki, students are inclined to 
revise more often. The quality of the product is increased, the student has practiced desirable behaviors (they did the 
assignment), and they are motivated to do more work, and better work in the future. 

The other important constructivist concept deals directly with the value of hypertext for learning. Spiro and others 
(Spiro, 1992) developed a theory of learning called Cognitive Flexibility Theory. Consistent with many other 
learning theories that conceive of knowledge as concepts that are individually stored in the brain as well as the 
relationships between those concepts. Cognitive Flexibility Theory suggests that learning is enhanced as many 
different pathways or relationships between stored concepts are established. A student would be more 
knowledgeable if he or she could link the same collection of concepts in many different ways. It may also be 
described that the student is more creative in his or her ability to work with that body of information. 

Spiro and his associates have also developed an instructional theory, called Random Access Instruction, 
that derives from Cognitive Flexibility Theory. They promote (Spiro, 1992) the use of hypertext environments, 
particularly in ill-structured domains, because students are able to explore many different paths through the same 
material. There is repetition of the content, but not replication because they take different paths and are likely to get 
different perspectives. 



Social Influences 

It is often difficult for writers to view their own work from other perspectives. That is one reason that the 
wiki is best used as a collaborative writing environment. The concept of collaboration has been interpreted 
differently in education. As many teachers do, I have put my students into small groups to work on particular 
projects with the expectation that all members of the group would contribute to the project in one way or another 
and with the expectation that they would all learn form the results of their combined efforts. What I have come to 
accept is that "combined efforts" are not often "collaboration." A group report on sharks is usually a collage of four 
separate reports on sharks pasted together. Ask one student about the report and they may answer, "I don't know! I 
did the part on what they eat!" 
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The wiki learning environment is very susceptible to co-authorship rather than collaboration. Whether the 
wiki responsibilities are shared by small groups or by whole classes, it is critical that the content and the 
organization of content is the result of social negotiation. This negotiation should also not be the kind that is typical 
of most peer groups when they are asked to review each other's work and give feedback. Most students are reluctant 
to critique others and seldom give specific feedback, if any at all (Soven, 1999). Thus, as Bednar et al point out 
(Bednar, 1992) "Sharing a workload or coming to a consensus is not the goal of collaboration; rather, it is to 
develop, compare, and understand multiple perspectives on an issue." (p. 28) Collaboration will be enhanced as 
group participants not only share knowledge, but use those multiple perspectives to construct meaningful artifacts 
(Resnick, 1996). In such a setting, "children view their peers as resources rather than as competitors." (Lunenburg, 
1998, p. 3) 



Learner Control - Let 'em Run 

It takes time for groups to exchange viewpoints and considerable more time for students to understand and 
appreciate the viewpoints of others, particularly if the students are not accustomed to group work in this way. 
Although they may require more supervision by the teacher at first, the collaborative process will begin to take over 
as the students become comfortable with each other and as they begin to honestly consider the thoughts and work 
contributed by each member of the group. As they gain confidence in their abilities to produce quality work, 
students also change their perception of their relationship with the teacher. Lehrer (1993) reported that students 
perceived themselves on a more equal footing with the teacher when they found the subject, in this case history, to 
be one that they discovered and interpreted themselves rather than waiting for the teacher to tell them what was true. 
"Teachers become participants in the students' activities and students become critical consumers of the teacher's 
activities." (Guzdial, n.d.). The reality is that students become more critical consumers of their own work as well— 
both product and process. 

Part of the reason that students rely less and less on the feedback from the teacher is that they begin to 
recognize "holes" in their understanding. The wiki environment makes it easy to identify, and to fix "holes" in the 
online products. At first it may be necessary for the teacher to review the students' hypertexts and to insert blank 
links whenever a concept needs to be further explored or where appropriate relationships between concepts have not 
been identified. A blank link in the wiki is indicated by a question mark after the linked term. They are easy for the 
author to find, but they do not interrupt the flow for the reader. As Leuf and Cunningham (2001) point out, there are 
no broken links in a wiki— only opportunities to add more content, (p. 22) It is important for students to learn to 
recognize holes themselves, and it is important that they learn to see those holes as "opportunities." Recognizing 
what a student does not know is valuable if they see it as a way to improve themselves rather than as a failure 
(Scardamalia, 1989). 

The more that students perceive that they have been given control over the learning environment, the more 
they will begin to recognize that the emerging product is a direct result of their efforts. Students assume ownership 
of the results of their efforts. Because the wiki environment is open to all— because anyone can change anything at 
any time— there are some important considerations for use as a learning tool. One recommendation is that it be 
closed to editing by anyone who has not registered as part of the group. This will reduce the likelihood that students' 
work will be negatively altered. Also, it is recommended that students be required to register with their real names. 
Anonymity may increase the contributions of some reluctant participants, but it is important that students attribute 
changes, be they better or worse, to a real person rather than to an unknown entity to which they have no recourse. 
The obvious implication is that students will probably not intentionally harm another student's work if their identity 
is clearly attached to their changes. 

As I conducted my own experiment of the wiki learning environment with two of my seventh-grade 
classes, I was particularly interested in the concept of ownership and how it would be affected as students, under 
assignment, edited and revised other student's wiki products — both within their own class and for students from the 
other class. I was very surprised to learn that most students were neutral regarding changes made by other students. 
However, one important trend did emerge. Students who felt that their pages were made worse through the revisions 
by others were highly possessive of their pages and were more critical overall of the wiki environment. Students 
who saw the revising as an improvement, and they were in the overwhelming majority, reported that the pages 
belonged to everyone and generally approved of the wiki experience. 

Implementation 

The physical establishment of a wiki learning environment is not overly difficult. It requires a special 
server and an Internet site. Most school districts have the hardware and expertise to make it happen, but they may be 
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reluctant to create the site and to pay for the URL because wikis are so uncommon and there are important issues 
about allowing students direct access to a district-hosted site.. Dr. David Wiley at Utah State University has been 
gracious enough to host our experimental site because of the research potential and I hope that the school district 
will be willing to assume responsibility for its maintenance when I can show them the value it has as a learning tool. 

Similar to the reluctance that may be found among district personnel, the major barrier to implementation 
within a school or a class is that it is a definite departure from what has traditionally gone on in secondary 
classrooms. I have experimented with the wiki learning environment in a classroom that has had a strong cognitivist 
and reception learning atmosphere. I found limited success for the majority of students, though there were a few 
who excelled within this new environment. As described earlier in this paper, the wiki is more suited to a 
constructivist approach. As Fosnot (1992) points out, the teacher is the key, "I have seen well-designed materials 
and instructional environments . . . totally misused by teachers who meant well but who held strongly to objectivist 
and transmission beliefs and thus interpreted the material's use through their model." (p. 175) The teachers are often 
uncomfortable with the constructivist requirements of the wiki experience and take back control of the learning 
process too quickly. (Simmons, 1993) Speaking directly about a wiki-like experience, Scardamalia et al. (1992) 
suggest that it requires an epistemological shift and not merely a methodological one. 

The students have also been found (Jonassen, 1992) to struggle with the constructivist approach if they are 
not used to it because they expect to be evaluated on their ability to spit back the correct answer rather than on their 
ability to work within the evolving environment and construct that environment as they do. Helping students to 
adjust to different learning styles will require patience and practice. Some students will make the transition better 
than others. Cullen (n.d.) found that the "more able" students seemed to get more out of this kind of activity. Lehrer 
(1993) recognized that the "more-successful" students were on task more often, but that the "less-successful" 
students gradually became more involved in the project. 

Conclusion 

Internet-based teaching methods are not new to the classroom, but they are only slowly coming into more 
general use as the availability of hardware improves in the schools. The ability and eagerness of educators to try 
these newer tools is also slowly improving as the influx of technology-trained teachers attacks the persistent school 
culture of the last century. The wiki learning environment may be just the tool that some of these forward-thinking 
teachers will employ because it is inexpensive and easy to use, and because it helps to facilitate the goal of creating 
motivated and independent learners. It remains to be seen if the social-learning and constructivist philosophies that 
are required to make it fully successful will be able to infiltrate the school culture as well. 
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The Virtual Instructional Designer: A Just In Time Instructional Design Tool 

Christine MacDonald 
Faye Bradshaw 
Indiana State University 

Abstract 

The Virtual Instructional Designer (VID), a web-based instructional design tool, is available for free and at 
your personal disposal 24 hours 7 days a week. The VID accommodates the needs of the novice and experienced 
online instructor in creating online or mixed modality courses. An example of how this tool was used to create an 
online developmental psychology course at Indiana State University will be demonstrated. 

Introduction 

The VID - The Virtual Instructional Designer (VID) is a Learning Anytime Anywhere Partnership (LAAP) 
grant-funded project from the Department of Education. The VID was created as a web-based performance tool to 
be used by post-secondary faculty for designing web-based distance courses. Instractional design teams at Indiana 
State University, Ivy Tech State College, and Vincennes University developed this project. Indiana State University 
serves as the lead institution and houses the VID project. The three-year grant for this project began in August 1999 
and ended August 2003. 

As web-based instruction continues to grow in popularity, the resources for developing web-based 
instruction have not increased to meet this demand. The Virtual Instructional Designer (VID) was developed to 
meet this demand and provide faculty with readily accessible and individualized instractional design assistance. 
Although this tool was designed for use by post-secondary faculty, instructional designers as well as teachers at the 
K-12 level can use the VID. 

The VID can be used to create a learning unit as well as an entire course. It may be used as an instructional 
design reference tool. The VID provides faculty and instructional designers with 24/7 desktop access instructional 
design assistance. This assistance includes the following: 

• Creating an effective course design and layout 

• Establishing goals and objectives 

• Linking goals and objectives to course content 

• Designing authentic assessments and assignments 

• Encouraging student engagement with course content 

• Incorporating multimedia into course content 

Material developed with the VID can be used by various courseware management systems including 
Blackboard and WebCT. More important, the VID is easy to use. There are no thick manuals to read or 
complicated instructions to follow. The VID, with its vast resources, is easy to navigate and use. 

Background 

In recent years, Indiana has been looking for innovative ways to reduce the number of students who fail to 
complete an associate or a bachelor’s degree. The Degree Link Program was established to resolve this problem. 

The Degree link partnership between Indiana State University (ISU), Vincennes University (VU), and Ivy Tech 
State College (Ivy Tech) allows traditionally underserved students to earn their associate degrees at partner 
institutions and then complete selected bachelor’s degrees at ISU. It was determined that the most effective way to 
offer degree programs and courses was through asynchronous, online courses. Partner institutions were now faced 
with the challenge of providing cost-effective, quality faculty training in the design and management of online 
courses. 
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Ms. Paula Vincini, former instructional designer and project leader at Indiana State University, was 
instrumental in developing plans for the Virtual Instructional Designer (VID). Moreover, Ms. Vincini was largely 
responsible for forging the relationship between partner institutions. Ivy Tech State College (Ivy Tech) and 
Vincennes University (VU), in the pilot testing and analysis of this project. 

The Project Team Included: 

• Grant Project Director 

• Project Leader (Instructional Designer) 

• Web Applications Programmer 

• Web Graphic Designer 

• Web Content Developers 

• Graduate Student Workers 

In addition, instructional designers (and faculty) from Ivy Tech State College and Vincennes University were 
involved throughout the development and pilot testing of the VID. 

Demonstration 



Access 

The Virtual Instructional Designer (VID) is a web-based tool that provides instructional design assistance to 
faculty and instructional designer. Here are the steps for accessing the VID: 

1 . Go to http://vid.indstate.edu 

2. Click “New User” 

3. Fill out the New User form 

4. Click Submit 

A username and password will be generated and sent to the email address that you included on the New User form. 

Design and Layout 

The VID is designed to provide optimal ease in navigating its more than five hundred pages of content. The 
Home page contains a page side navigation bar, top navigation bar, and a main window. The page side navigation 
bar allows you to access the seven major content areas of the VID: Jump Start, Communicate, Content & 
Assignments, Motivation, Evaluation, Media & Technology, and Tutorials. The top navigation bar includes a 
search, glossary, FAQs, site map, and tour (orientation area). A My Features area, located at the top right of the 
page, contains an area for taking notes, creating a VID plan for course design, and a bookmark function for marking 
pages of interest. 

Content 

Content within the VID includes best practices, examples from faculty experiences, current research, 
instructional design advice, and information regarding media and technological advances. This content is divided 
into seven areas: Jump Start, Communicate, Content & Assignments, Motivation, Evaluation, Media & 
Technology, and Tutorials. 

Jump Start provides a "big picture" about online courses as well as how to get started designing 
instructional activities. Communicate provides information on how to design quality discussions. Content and 
Assignments gives information on how to create and adapt content for online use. Motivation provides insightful 
information on how to create and maintain an online learning community. Evaluation covers various topics such as 
designing authentic assessments and evaluating groups. Media & Technology addresses learning style issues related 
to the incorporation of audio and video components in online courses. Tutorials provide important information on 
how-to skills for developing instruction such as digitizing course material or transforming documents into portable 
document files (PDF). 

How to Use the VH) 

The Virtual Instructional Designer (VID) is designed to assist you with at least three types of instructional 
design assistance: 
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1 . 

2 . 

3. 



Creating online course components (e.g., syllabus, lecture notes, assignments and assessments). 

Exploring specialized topics (e.g., learning communities, experiential learning, and learning styles) 
Planning and revising a course (e.g. inventorying, sequencing, and incorporating course 
material/multimedia). 

Use the search function or click on a content module to find information about online course components or 
specialized topics. If you are planning and/or revising a course, there are two areas of the VID that you may use; 

My Features and Jump Start (Managing the Design Process). Click on My Features drop down menu and select My 
VID Plan. The My VID Plan link will take you to the Faculty Needs Survey (Figure 1). Select items in this survey 
that pertain to your course needs, and click Submit. The survey generates a learning plan consisting of links to 
pages in the VID that you can read and take notes on (Figure 2). Next, go to Jump Start (content module) and click 
on Managing the Design Process. This section includes a step-by-step process for planning your course (Figure 3). 

Here are some sample artifacts of the design process. 

Figure I. Faculty Needs Survey 



Faculty Needs Survey 

The following survey will help you decide what areas of interest and information you need to pursue in 
the Virtual Instructional Designer. The survey will generate a Web page of links to VID pages 
pertinent to that category called My VID Plan. You can access My VID Plan in the left frame from 
anywhere in the VID. 

1. Beginning the Virtual Design Project 

A. Ensuring Quality in Instruction 

r* Meeting institutional quality benchmark 
r~ Retaining the quality of your instruction 

I” What learners, instructors, and researchers say about online quality 



Figure 2. My VID Plan 



VID Plan for "Developmental 
Psychology" 

1. Beginning the Virtual Design Project 

A. Ensuring Quality in Instruction 

• Retaining the quality of vour instruction 

• What learners, instructors, and researchers 
say about online quality 

B. Ensuring Quality in Design 

• Basic yyeb page design rules 

• Best practices in yyeb pane design 
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Figure 3-1. Managing the Design Process 



Managing the Design Process 

YouVe just agreed to transform your face-to-face course for online delivery 
a year from this semester, or you may simply want to enhance your 
traditional course with web-based activities. 

Either way, you must begin to think of yourself as a Project Manager 
because of the complexity of designing these interactive, learner-centered, 
technology-based learning environments 

You will need a systematic, organized pian to manage your design and development process and 
one way is to use the following four step process for creating a Course Design Blueprint. 

Even if you only plan to introduce Web-based groups or discussions to your course, these steps can 
be modified to create a solid design blueprint to follow. 




Figure 3-2. Managing the Design Process 



^ Creating a Course Design Blueprint 



Step 1 : Inventory Your Course Activities 

Look at the face-to-face course and create a Course Inventory Grid of what happens with and to 
students throughout a typical semester. Models are provided for you to use or revise. 



Step 2: Analyze Communication Strategies and Your Learner's Goals 

Now that you have an inventory of course activities, assignments, and communication, take 
some time to analyze course communications and the goals and characteristics of your online 
learners. What potential challenges appear in terms of continuing to use traditional classroom 
methods? 
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Figure 3-3. Managing the Design Process 




Step 3: Analyze Course Learning Goals & Learning Objectives 



Because the goals and objectives for your course will determine the basic design framework for 
the instructional strategies you will use, the content you will present, and the nature of your 
evaluations, create a final G & 0 grid. This provides the beginning for your blueprint. 



Let's begin by taking a short 'quiz ' on goals and objectives. 

Course Learning Goals and Learning Objectives 

• Writing Clear and Effective Learning Goals 

• Writing Clear and Effective Learning Objectives 



Goals & Objectives Grid 

• Bloom's Cognitive Taxonomy 

• Developmental Psychology Goals & Objective Grid 

• Nursing Leadship Goals & Objective Grid 



Application to Course Development 




Indiana State University 










Start Here Course Documents Calendar Mail 







Discussions Exams and Surveys Student Tools Group Projects 







Course Content Overview Group Project drop box 
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The VID can be an invaluable tool for developing online courses, as it was in developing this online 
undergraduate developmental psychology course. Using the VID was an eye-opening experience in terms of how to 
help students succeed in this course from the very beginning. The section on designing a course orientation (under 
"Jump Start") was particularly helpful. For example, in order to make students familiar with the course site, the VID 
suggested creating a "scavenger hunt" quiz (see Figure 4). This involved putting together a list of questions about 
the course, such as: "How many practice quizzes are you required to complete?" and "Find the names of two other 
students enrolled in the course". In order to successfully complete the scavenger hunt, students must navigate 
different parts of the course site. In this course, students can find answers by finding the syllabus, the calendar, the 
instructor's website, the class roster, and the netiquette guidelines. The other advantage to the scavenger hunt is that 
it allows students to practice taking a quiz online. 



Figure 4. Scavenger Hunt 



^tart Here 


o 


i® 


Important! Read this First. 


Syllabus 


/f 




Learner Analysis 


Top 10 Hints 




^llabus 


Scavenger Hunt 


Syllabus •• Word Format 



Another suggestion from the VID that was incorporated into this class was the use of a learner analysis (see 
Figure 4). This survey asks students for information not only about their technical skills and previous experience 
with course content and distance education courses, but also about their expectations for the course and some basic 
demographics. The learner analysis serves several functions. It provides information relevant to student’s chances 
of success in the course and the types of support that they may need. The demographic information collected 
(gender, age, major, number and age of children, if any) is used to divide students into groups for their group 
projects in such a way as to allow for maximum diversity of viewpoints on development within each group. 
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Figure 5. Learner Analysis (partial view) 




A particularly helpful section of the VID is on the behaviors of successful online students and how to 
encourage students to engage in these behaviors. One of the characteristics of successful online students is that they 
monitor their progress in the course. Therefore, multiple supports have been provided for students to help them do 
this. These include making students aware of the accessibility of the online gradebook (and letting them know how 
often grades will be updated) as well as their “attendance” records. Since students are required to perform a number 
of activities each week (read assigned chapters, read the week’s online content overview, participate in discussions, 
take practice quizzes, and work on group projects), a checklist of each week’s activities was also created to keep 
them on track (see Figure 6). 

Figure 6. Checklist of Weekly Activities 



Course Documents 




IS 




IS 

syllabus 


Syllabus 


Syllabus •- Word Format 








Communication Guidelines 




Checklist 



The “Content & Assignments” section of the VID contains useful information about learning activities and 
how to best design them for online courses. For example, the module on lectures requires the faculty member to do 
very thoughtful analysis of why they want to post lecture notes online, what those notes should look like, and how 
online lecture notes might differ from traditional lectures. This analysis led to the conclusion that if the textbook did 
a good enough job of explaining the content, there was no real need for that content to be replicated online. One of 
the goals of this course was for students to be able to interact with the content at a deeper level and be able to apply 
it to real life situations. Thus, lecture notes were not posted. Instead, students respond to a series of asynchronous 
threaded discussions throughout the semester (see Figure 7). 
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Figure 7. Threaded Discussions 



Discussions 



Discussions 



1 Display: ^All I^Unreadl ^ >^Threaded f Unthreaded] Select tooic: 1 Week l 




Status r Subject Author 


Date 



' 0/1 r ^ chapter 1 - Question 1 
Q r Chapter 1 • Question 1 



Christine MacDonald (EPSY221) Jure 13, 2002 3;27pm 



Subject: Chapter 1 - Question 1 
Message no. 1 

Author: Christine MacDonald (EPSY221) 

Date: Tuesday, June 18, 2002 3;27pm 
Given thac children ere among che poorest groups in 
country, and that poverty is associated with increa 
risks throughout the lifespan, how should we helps ' 
children and/or their families? 



These discussions involve critical thinking about issues related to material from that week’s reading, and 
involve integration of text material and life experiences with case studies or real life issues and events. The 
“Designing Threaded Discussions” unit of the VID (under “Communicate”) was used to design the evaluation 
rubrics and criteria associated with these discussions. Students have available to them online a list of criteria used to 
evaluate their weekly discussion contribution, as well as specific examples of graded discussion contributions. Each 
example also includes a rationale for the grade assigned. The VID also includes a statement on how much the final 
course grade course discussions should count for, in order to assure student participation. 

Summary 

The Virtual Instructional Designer, developed as a part of a Department of Education grant, is a web-based 
instructional design tool that can be used to develop online courses. The VID was developed to assist the Degree 
Link Partners, IVY Tech State College and Vincennes University, in developing online courses for underserved 
students who wish to complete their bachelor's degrees at Indiana State University. 

The Virtual Instructional Designer (VID) is designed to assist you with at least three types of instructional 
design assistance; 

1. Creating online course components (e.g., syllabus, lecture notes, assignments and assessments). 

2. Exploring specialized topics (e.g., learning communities, experiential learning, and learning styles) 

3. Planning and revising a course (e.g. inventorying, sequencing, and incorporating course 
material/multimedia). 



An example of how this tool can be used is illustrated by the design of a Development Psychology Course 
(EPSY 221) at Indiana State University. Sections of this tool. Jump Start, Contents & Assignments, and 
Communications, were used to create effective course components including: orientation/scavenger hunt, learner 
analysis, various tools for students to monitor their progress, and threaded discussions. 
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XML Primer and eLearning Applications 



Paul F. Merrill 
Brigham Young University 

Abstract 

XML (extensible Markup Language) is one of the latest buzzwords. What is it? How is it related to 
HTML? Is it worth learning? Is it hard to learn? How can it be used in eLearning applications? This presentation 
will address each of these questions and more. It will include a description of a simple tutorial application that 
shows the power of using XML to separate content, presentation specifications and interaction strategies. 

Introduction 

XML is an acronym for “extensible Markup Language.” XML provides a mechanism for marking or 
tagging information or data. The marks or tags serve as metadata, or data about the data. These descriptive tags can 
provide clues as to the structure and meaning of the data. Let’s examine the simple well-formed XML document 
shown in Figure 1 . 



<?xml version="L0"?> 

<library> 

<book> 

<title>Instructional Design, 2nd Edition</title> 

<authors> 

<author>Patricia L. Smith</author> 

<author>Tillman J. Ragan</author> 

</authors> 

<copyright> 1 999</copyright> 

<publisher>John Wiley and Sons, Inc.</publisher> 

<cover src="coverArt.jpg"/> 

</book> 

<book> 

<title>XML for the World Wide Web</title> 

<authors> 

<author>Elizabeth Castro</author> 

</authors> 

<copyright>200 1 </copyright> 

<publisher>Peachpit Press</publisher> 

<cover src="XMLCover.jpg"/> 

</book> 

</library> 

Eigure 1 . A well-formed XML Document 

As shown in Eigure 1, the data or content is divided into elements that are enclosed within opening and 
closing markup tags. The metadata markup tags are emphasized in bold. Thus, the content, “Instructional Design,” 
is enclosed within the tags, <title> and </title>. XML opening tags begin with the symbol < and end with the 
symbol >, while closing tags begin with <J and end with >. Note that the <author> tags are nested inside of the 
<authors> tags, which are nested in turn inside of the <book> tags. This nesting of tags organizes the content 
elements into a hierarchical structure. Tags that contain no content, such as <cover src="XMLCover.jpg"/>, begin 
with the symbol < and end with />. Embedded within the <cover> tag is an attribute, src, and it’s corresponding 
value, "XMLCover.jpg”. 

The example XML document shown in Eigure 1 is said to be “well-formed” because it conforms to a few 
rules outlined in the XML specification such as the following: 
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1 . 



Each XML element has matching opening and closing tags. Empty tags that contain no content may 
use the shorthand notation shown with the cover tag in Eigure 1. 

2. The XML document has one root element within which all other elements are nested. 

3. All elements are properly nested 

4. Attribute values are enclosed in quotes. 

One of the reasons XML is referred to as “extensible” is because the labels that make up the tags are not 
predefined. Authors of XML documents can use any tag labels they desire. The tags have no inherent meaning of 
themselves. Meaning is imposed on them by the author. Thus, the author should use care in choosing tag labels that 
provide meaningful descriptions of the data. 

How is XML related to HTML? 

HTML and XML are both based on the more complex Standard Generalized Markup Language (SGML). 
XML is a simpler subset of SGML, while HTML is an application of SGML (Llynn, nd). HTML (Hypertext 
Markup Language) is used to markup data to indicate how it should be displayed on a web page. In contrast, XML 
is used to indicate the meaning and structure of the data. XML was not necessarily designed to replace HTML. 
XML and HTML serve different functions (XML Tutorial, nd). To highlight these differences, let’s examine a 
simple HTML document shown in Ligure 2. 



<HTML> 

<HEAD> 

<TITLE>Simple HTML Example</TITLE> 

</HEAD> 

<BODY> 

<H 1 >Books</H 1 > 

<HR> <BR> 

<B>Title:</B> Instructional Design 
<BRxB>Authors</B> 

<UL> 

<LI>Patricia L. Smith 
<LI>Tillman J. Ragan 

</UL> 

<IMG SRC="coverArt.)pg"> 

</BODY> 

</HTML> 

Ligure 2. Simple HTML document 

As can be seen by comparing Ligure 1 and Ligure 2, there are many similarities between XML and HTML. 
Both documents contain content along with markup tags. The XML tags describe the content, while the HTML tags 
often specify how the content is to be displayed. Lor example, <B>Title </B> from Ligure 2, specifies that the word 
“Title:” should be displayed in boldface type. The <HR> tag specifies that a line or horizontal rule should be 
displayed on the screen. The <UL> tag indicates that the following items should be displayed as a bulleted (or 
unordered) list. Some HTML tags such as the <H1> tag serve duel purposes. <H1> indicates that the embedded 
content is a level one heading, but also indicates that the heading is to be displayed in large boldface type preceded 
and followed by a blank line. 



In contrast with HTML, the XML document shown in Ligure 1 contains no information on how the 
information or data is to be displayed. This is both an advantage and a disadvantage. The disadvantage is that the 
way the XML document is to be displayed in a web browser must be specified by a separate style sheet document. 
The advantage is that the content or data is separated from the presentation specification. This allows us to modify 
the style sheet to change the way the data is displayed without modifying the data. We can also use the same style 
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sheet for different sets of data. It also allows us to use the XML data for multiple purposes. Style sheets will be 
discussed in more detail in a subsequent section. 



Why XML? 

Although XML documents with meaningful tags are easily interpreted by humans, their great value lies in 
the fact that their formal structure also makes it possible for them to be easily interpreted by computers. This makes 
XML documents ideal for the interchange of information or data over the Internet (Bryan, 1997). Because XML 
documents can be stored in standard text files, and they are independent of platforms, hardware, operating systems, 
software or applications, they can by used and processed by a wide variety of applications for many different 
purposes (XML Tutorial, nd). XML is already being used for many applications Harold (2001). 

For example, the Chemical Markup Language (CML) is an XML application used to represent complex 
chemical data such as molecular structures. Mathematical equations can be represented using Mathematical Markup 
Language (MathML) and sheet music can be represented using the XML application MusicML. Two-dimensional 
vector graphics can be described using other XML applications such as Scalable Vector Graphics (SVG) or the 
Vector Markup Language (VML). Figure 3 shows the XML for an SVG document describing several vector 
graphic objects such as a circle, rectangle, ellipse, line, and a polygon. 



<?xml version="L0"?> 

<svg xmlns="http://www.w3.org/2000/svg"> 

<circle r="30" cx="35" cy="35" fill ="red" stroke = "blue" stroke-width = "2" /> 

<rect X = "95" y = "10" width ="30" height = "50" rx = "5" ry = "10"/> 

<ellipse cx = "200" cy = "35" rx = "50" ry = "20" fill = "gray'7> 

<line xl = "15" yl = "100" x2 = "75" y2 = "150" stroke-width = "2" stroke = "green"/> 

<polygon points = "100,100 100,150 150,150" stroke-width = "2" stroke = "blue" fill = "yellow"/> 
</svg> 

Figure 3. XML document using Scalable Vector Graphics (SVG). 

Jon Bosak (1999) has marked up all of Shakespeare’s plays and four religious texts (the Old Testament, the New 
Testament, the Koran and the Book of Mormon) into XML. Figure 4 shows a portion of the XML document for 
Shakespeare’s Hamlet. 
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<?xml version="1.0"?> 

<PLAY> 

<TITLE>The Tragedy of Hamlet, Prince of Denmark</TITLE> 

<PERSONAE> 

<PERSONA>CLAUDIUS, king of Denmark. </PERSONA> 

<PERSONA>HAMLET, son to the late, and nephew to the present king.</PERSONA> 

</PERSONAE> 

<ACT> 

<TITLE>ACT III</TITLE> 

<SCENE> 

<TITLE>SCENE I. A room in the castle.</TITLE> 

<STAGEDIR>Exeunt KING CLAUDIUS and POLONIUS</STAGEDIR> 

<STAGEDIR>Enter HAMLET</STAGEDIR> 

<SPEECH> 

<SPEAKER>HAMLET</SPEAKER> 

<LINE>To be, or not to be: that is the question:</LINE> 

<LINE> Whether 'tis nobler in the mind to suffer</LINE> 

<LINE>The slings and arrows of outrageous fortune,</LINE> 

</SPEECH> 

</SCENE> 

</ACT> 

</PLAY> 

Eigure 4. Portion of Shakespeare’s Hamlet in XML 

As Harold (2001) suggests, the advantage of marking up Shakespeare’s plays in XML is that it makes it 
possible for a computer to analyze and manipulate the text in a variety of ways. Eor example, a version of a play 
could be printed which highlights the lines of a particular character in bold face. 

Later in this paper, I will show how XML can be used to represent content in e-leaming applications. But, 
first, let’s see how style sheets can be used with XML to specify how the XML content should be presented in a web 
browser. 

What is a style sheet and how is it used with XML? 

Most pages on the Web are coded in HTML. As mentioned earlier, these HTML documents combine the 
content and presentation specifications in one document. In contrast, XML documents usually do not contain any 
presentation specifications. If you want to display the content of an XML document in a browser, you can prepare a 
separate style sheet that provides the corresponding presentation specifications. Ligure 5 presents an example of a 
style sheet that specifies how the Library XML document shown in Ligure 1 might be displayed. 

library {font-family: Palatino, Garamond, Times, serif; margin:20pt} 

title, author { display: block; margin-bottom:! Opt} 

title (font-size: 18pt; color: red; font-style: italic; font- weight: bold} 

author (font-size: 14pt; text-indent: .25in} 

copyright, publisher (display: none} 

Ligure 5. Style Sheet for the Library XML Document 

The style sheet shown in Ligure 5 consists of a set of five rules for how to display various elements of an 
XML document. These rules conform to the Cascading Style Sheet (CSS) specification 
(http://www.w3.org/Style/CSS/) . 
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Each rule begins with the name or tag of the XML element to which the rule should be applied. The name 
is followed by a list of styles, within brackets, that are to be applied to elements with that name or tag. Thus, the first 
rule specifies that all of the content nested within the root library tag should be displayed with a serif font and within 
a margin of 20 points. The second rule specifies that the contents of a title or an author tag should be displayed in a 
separate block or line with a bottom margin of 10 points. The third and fourth rules provide additional styles that 
are to be applied to the title and author tags respectively. The fifth rule indicates that the content of the copyright and 
publisher tags should not be displayed. 

These rules would be saved in a separate file with a .css suffix, such as “library.css”. The second line of 
the corresponding XML document should contain a processing instruction that would instruct a Web browser to use 
this style sheet to determine how to display the XML content: <?xml-stylesheet href="library.css" type="text/css"?>. 
Ligure 6 shows a screen shot of how the XML document shown in Ligure 1 would be displayed using the style sheet 
from Ligure 5. 



InstriKtiouiil DcAigiiy 2iui Editio3i 
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Ligure 6. Screen shot of XML document using style sheet from Ligure 5 

Simply simply changing the style sheet can dramatically modify the look and feel of the XML document. 
Ligure 7 shows an alternate presentation of the same XML document using a different style sheet. 



instructionai Design, 2nd Edition 



L. Sniuth 
Tillmsfl J. .ftagan 



XML for the Wide Web 
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Ligure 7 Screen shot of the same XML document using an alternate style sheet 

The real power of style sheets becomes apparent when a single style sheet is applied to multiple 
documents. A change to the common style sheet will automatically result in a change to all of the corresponding 
documents. 

How can XML be used in elearning applications? 

Many elearning applications available on the Web are also coded in HTML. Simple HTML elearning documents 
are generally static and just provide information. More sophisticated elearning applications include interaction. 
Students are asked to respond to practice situations and corrective feedback is provided for their responses. Such 
interactions are made possible by adding computer-programming code to the Web page using a scripting language 
such as JavaScript. This approach combines the content, presentation specification and interaction code into the 
same document. 
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By placing the content in an XML document, it is possible to separate the content, the presentation 
specification, and the interaction script. This separation facilitates maintenance and reuse of all three separate 
elements. 

Let’s now look at a simple tutorial for teaching the concept of “adjectives” that demonstrates how this 
separation could be implemented in an elearning application. 

Figure 8, 9 and 10 show screen shots of three pages from the tutorial. Figure 8 shows a page that presents a 
definition of the concept. Figure 9 shows one of several pages that present examples of the concept, while Figure 10 
shows one of several pages that provide students with an opportunity to practice identifying the adjective in an 
unencountered sentence. 



An Adjective is a 
word that 

modifies a noun or a 
pronoun 



Figure 8. Screen shot of the concept definition page. 



1 want some apple 
pie. 

Apple is an adjective 
since it modifies pie. 

rHult'll I PlKlitt'll Qjf I 



Figure 9. Screen shot of an example page. 
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The $ky is sure blue. 




Figure 10. Screen shot of a practice page. 

The text of the examples and practice items are not hard coded in HTML. Instead they are stored in external 
XML documents. Figure 1 1 shows the XML document for the example items, while Figure 12 shows the XML 
document for the practice items. 



<?xml version="L0"?> 

<EXAMPLESET> 

<EXAMPLE> 

<SENTENCE>1 want some apple pie.</SENTENCE> 

<COMMENT> Apple is an adjective, since it modifies pie.</COMMENT> 

</EXAMPLE> 

<EXAMPLE> 

<SENTENCE> Where is my red pencil.</SENTENCE> 

<COMMENT>Red is an adjective, since it modifies pencil. </COMMENT> 
</EXAMPLE> 

<EXAMPLE> 

<SENTENCE>This is a wonderful conference.</SENTENCE> 

<COMMENT>Wonderful is an adjective, because it modifies conference. </COMMENT> 
</EXAMPLE> 

</EXAMPLESET> 

Eigure 1 1 . XML document for Example items 
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<?xml version="1.0"?> 

<PRACTICESET> 

<PRACTICE> 

<ITEM>Jane has a sports car.</ITEM> 

<PART WORD ="Jane">noun</PART> 

<PART WORD ="has">verb</PART> 

<PART WORD ="a">article</PART> 

<PART WORD ="sports">adjective</PART> 

<PART WORD ="car">noun</PART> 

</PRACTICE> 

<PRACTICE> 

<ITEM>The sky is sure blue.</ITEM> 

<PART WORD ="The">article</PART> 

<PART WORD ="sky">noun</PART> 

<PART WORD ="is">verb</PART> 

<PART WORD ="sure">adverb</PART> 

<PART WORD ="blue">adjective</PART> 

</PRACTICE> 

<PRACTICE> 

<ITEM>That is a pretty dress. </ITEM> 

<PART WORD ="That">pronoun</PART> 

<PART WORD ="is">verb</PART> 

<PART WORD ="a">article</PART> 

<PART WORD ="pretty">adjective</PART> 

<PART WORD ="dress">noun</PART> 

</PRACTICE> 

</PRACTICESET> 

Eigure 12. XML document for Practice Items. 

The XML document in Eigure 1 1 contains tags for the sentences and comments that will be displayed when 
the student clicks on the Example button. The XML document in Eigure 12 contains tags for sentence practice items 
and tags that identify the part of speech for each word in the practice sentences. These part-of-speech tags make it 
possible for the computer software or interactive script to construct unique, informative feedback messages for any 
anticipated student response. 

The content of the examples and practice items can be modified, or additional items can be added without 
changing one line of the computer program or script that controls the interaction and feedback. On the other hand, 
the XML practice document can be reused without modification to teach the concepts of “nouns” or “verbs” by only 
changing the value of a single variable in the computer script for teaching “adjectives.” Eigure 13 shows the screen 
shot from the modified tutorial that uses the same XML practice document to present “noun” practice items. 
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Figure 13. Screen shot of “noun” practice item. 

In addition the tutorial uses an external style sheet to control the format of the presentation. By making 
simple changes to the style sheet, the background image or color and font characteristics of the tutorial pages can be 
modified without changing the HTML, interaction script, or XML. An example of the effects of different style 
sheets can be seen by comparing Figures 10 and 13. 

Thus, by using XML to markup the content materials and style sheets to specify the presentation format, 
the reuse of the content and the interaction script is facilitated. In fact, the tutorial can be replicated using other 
authoring tools such as Flash or iShell, which can read the same XML files without modification. 

Summary 

This paper has shown how XML (extensible Markup Language) can be used to markup or tag content. 
Since XML documents, with meaningful tags, can be easily interpreted by humans as well as computers, they are 
ideal for the interchange of information. XML documents have proven useful in a variety of areas including 
chemistry, mathematics, music, literature, graphics, and eCommerce. This paper demonstrated how XML could be 
used as a tool for developing eLearning applications where the content, presentation specification and the interaction 
script can all be separated from one another. This separation facilitates maintenance and reuse of all three separate 
elements. 
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Ethical Issues In Copyright Compliance and Fair Use Guidelines in Teacher 

Education 

Vicki S. Napper 
Weber State University 

Abstract 

There is an alarming lack of awareness of both copyright law and fair use guidelines in educational 
setting. Students at advanced stages of secondary education frequently do not make the connection between 
plagiarism and abuse of copy rights, and public schools do not appear to be actively promoting copyright 
compliance by students or teachers. Fair use guidelines are also not readily understood by licensed educators and 
seldom known by students. Multiple issues fuel the breaking of copyright compliance: (1) lack of funding to 
purchase materials, (2) lack of understanding of the economic benefits of copyright, and (3) lack of awareness of the 
fair use guidelines. Although fair use guidelines are provided for educators and students there still remain the 
unresolved issues of protection of intellectual properties of teachers and students as well as availability free, quality 
resources to promote educational reform. 

Why, in a society rich with information and an exponentially expanding reservoirs of accessible resources, 
our public educators the least likely to understand how to access and use these materials and the least likely to pass 
on an awareness of copyright laws to their students? This should not be a debatable issue; however, I have found 
this to be true in many of the educational settings I have worked in as well as studied in. Now as faculty in a teacher 
education program, I am again surprised at the lack of awareness of copyright laws by students entering my program 
and also at the lack of compliance shown within educational settings they have emerged from. This discussion is an 
attempt on my part to find solutions to this issue for my students and my colleagues. 

Problems 

Low Levels Of Literacy About Copyright 

After analyzing the content of response papers submitted by students over a two-semester period of time, I 
discovered that more than 50% of the students did not know that copyright law applied to educators. There was a 
genuine response of surprise. In light of the fact that all of the students were either at the junior level of college or 
had already received degrees and were returning to obtain licensure for teaching, this was surprising to me. Many of 
the students expressed the idea that even through they were aware of copyright law; they did not think copyright law 
applied to teachers and that teachers were free to use any materials available to them in any manner they desired. 

This is a very disturbing idea to emerge out of any population of students. These students were new into the teacher 
education program and had no or little exposure to actual courses within the program. The conclusion is 
inescapable: the public education system had not created environments that promote the idea of protection of 
intellectual properties nor of the right of the creator of products to control access to those products. 

“I knew there were such laws, but I was under the impression that the enforcers of the laws 
skipped over us or turned their heads because we were educators and the information was there for 
us to educate the youth of this great nation.” (JM) 

“I never knew there were so many limitations when incorporating copyrighted materials into your 
own presentations and works. I always thought that I just needed to cite my sources to be all right. 

Most of my classes taught me what plagiarizing was and how to avoid it, but I never learned about 
copyright laws for the use of multimedia products.” (CD) 

These statements and sentiments were echoed throughout the responses from five sections of 
entry-level technology students. Many of the students made the connection between plagiarizing and 
copyright laws but some did not. Most were surprised by the complexity of copyright law and fair use 
guidelines. Many were discouraged by copyright protections and many others were happy to find the fair 
use guidelines outlined by the Consortium of College and University Media Centers' 

Perhaps as disturbing but not unexpected was the finding that most of the school districts contacted in a 
survey did not have any formal way to promote copyright compliance. Research was conducted on six school 
districts surrounding the university. The school districts are both urban and rural. The three urban districts have a 
total student population of over 101,000. The three rural districts have a total student population of a little over 
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16,250. The largest urban district has 60,000 students, and the smallest rural district has approximately 1,000 
students.^ 

The survey included the following questions: Does the district have a person(s) serving as a technology 
specialist(s) who informs teachers and students about: (1) First Amendment protection, (2) filtering and blocking 
obscene material, (3) use of students’ personal information and education records, (4) downloading and storing 
copyrighted material, (5) fair use, (6) defamation in Internet communications, (7) and E-mail harassment? 

Each of the six districts had a person designated as a technology specialist who has responsibility 
for First Amendment protection, filtering and blocking obscene material, proper use of students ’ 
personal information and education records. Each specialist was also responsible for the correct 
downloading and storing of copyrighted materials, fair use practices, defamation in Internet 
communications and E-mail harassment. None of the six districts reported that they had addressed 
First Amendment protection issues, defamation in Internet communications or E-mail 
harassmen.^t 

Additional questions in the survey included ones related to policy on copyright liability, training for new 
teachers. Codes of Ethics/Conduct, and signs warning users of the misuse of materials. These were the findings: 

• Three of six districts had a board policy on copyright liability. 

• Each of the six districts had an acceptable use policy. Three districts out of six train new 
employees on copyright issues and acceptable use policies. The largest urban district reported 
that their technology personnel train others from a book, but the training usually consists in just 
answering questions when a problem arises rather than any formal training. They also reported 
that their Code of Conduct book is very general. 

• No district had any acceptable use policies in a code of ethics/conduct manual for teachers or for 
students. Only one district had acceptable use policies in their Teacher/Administrator and 
Classified Employee manual. One district had a brochure on acceptable use policies. 

• District personnel were unsure whether there were posters around copy machines in their various 
schools where the misuse of copyright laws were posted."* 

It is a reasonable question to ask that if indeed copyright laws were being complied with 
consistently within educational environments, why were only half of the school districts creating any policy 
to guide compliance? And conversely, if an awareness of copyright laws were pervading the school 
districts thereby not requiring any kind of board policy to enforce compliance to a Eederal Law, why are 
there so many students who are unaware of the application copyright laws in educational settings? These 
are not easy questions to answer. 

No Levels Of Literacy About Fair Use Guidelines 

One possible answer to the question about copyright awareness is that teachers do not understand there are 
fair use exceptions made to Copyright Law for educators and students. In environments where resources are thin or 
non-existent, the urge to use materials regardless of unforeseeable consequences can be high. This is not to imply 
that educators are not moral about what they do; instead, it could easily be said this is a group of people who are 
working against many obstacles (economic as well as political) to provide adequate and timely information to 
students. Anyone who has ever been a teacher can recall finding just “the perfect” resource to use in a class and not 
being able to purchase enough copies for use by students because there was no budget for extras. The option usually 
is to have multiple students share one or few resources. In places with high numbers of students, this can be 
problematic in the flow of instruction. How many class periods should a student wait until they have access to the 
information or resource for learning? In an ideal world, none. The materials should be available when needed 
without the need to replicate illegal additional sets. 

Frustration About Lack Of Available Materials For Use In Classrooms 

When the above mentioned entry level students into teacher education were first introduced to the idea of 
the fair use exclusions, they either expressed relief at having a way to use materials or confusion about the materials 
provided to them by school districts or hostility at a limitation to their ability to teach with current or appropriate 
materials they did not believe would be provided in a timely manner to them or their students. This is a question that 
is not easy to solve. Certainly under the principle of copyright protection, the creator of products deserves to control 
the monetary rewards of their efforts. The irony is that many of the authors of books or articles actually loose control 
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over the dissemination of their ideas. They cannot give permission to a teacher to use those materials, but instead, 
must refer the teacher or student requesting permission to use copyright compliance personnel in large corporations. 
There should be a streamlined process outlined and readily available to exactly the purpose of requesting permission 
to use materials. Most teachers and even fewer students know how to obtain permission for use of materials. The 
assumption being made about using materials without permission is if “they” don’t know, it won’t hurt anyone. 

Solutions 

Awareness Of The Economic Benefits Of Copyright Compliance 

It is extremely important to make a connection between an economy based on information and the financial 
benefits of the copyright process. Giving the response that copyright is the law of the land does not inspire 
compliance. There should also be motivating reasons for compliance. Frequently in discussions with teachers and 
students, they comment on how much money the copyright holder must be making compared to the end users’ 
income base. What is frequently not mentioned is the economic base created by the business of replication of 
copyrighted materials. It is easy to loose site of the idea of people other than the copyright holder or Big Businesses 
benefiting from copyright compliance. The idea of people depending on the sale of copyrighted materials for the 
continuation of their jobs creates support of copyright benefits. The growth of the economic structure of an 
information rich society cannot afford societal ignorance of what factors create that structure. 

Awareness Of Fair Use Protections For Educators 

The umbrella of fair use provides benefits for educators and students. The restrictions for time are more 
restrictive for teachers who are displaying or using copyrighted materials than for students, but both may use 
copyrighted materials, in proper proportions, to extend their ability to teach and to learn and for proof of their 
talents. This is the intended focus of the educational system. The ability to access ideas is not limited. Ideas cannot 
be copyrighted. The uniqueness of expression of an idea is what allows the process of continual, intellectual, 
creative renewal to occur and thrive. 

If we are indeed in a culture based on the cumulative nature of information as well as the expression of new 
ideas arising from problem-solving processes, the body of professional educators should not be reusing fixed ideas 
without renewal. Understanding of knowledge evolves and within that context of understanding, people change in 
their perceptions and processes of applying knowledge. Fair use not only protects the accrual of copyright benefit to 
rightful owners but it also provides protection to the intellectual integrity of this nation through the promotion of 
change. Specifically, the fair use guidelines promote appropriate usage of multimedia presentations, rich with 
knowledge and expression of ideas, for two years. Then the same fair use guidelines promote the idea of renewal 
and change (Fair Use Guidelines, Section 4, time, portion, copying and distribution). According to the American 
Library Association; (1995, Fair Use In The Electronic Age section, f) 

The genius of United States copyright law is that, in conformance with its constitutional foundation, it 
balances the intellectual property interests of authors, publishers and copyright owners with society's need 
for the free exchange of ideas. Taken together, fair use and other public rights to utilize copyrighted works, 
as confirmed in the Copyright Act of 1976, constitute indispensable legal doctrines for promoting the 
dissemination of knowledge, while ensuring authors, publishers and copyright owners appropriate 
protection of their creative works and economic investments. ^ 

A Reflection On Application Of Copyright And Fair Use In Actual Settings 

Although multiple reasons for compliance with copyright laws and fair use guidelines can be made, the 
final resolution will ultimately be determined by availability of resources. The pressure of education is to provide 
learning experiences. The pressure of copyright is to provide income. This issue will continue to create conflicts. 
The resulting balance must be a resolve for those groups depending on the flow of information for educational 
purposed to find solutions for the future. If this balance is allowed to decay or become lost, there will be a huge loss 
in personal educational freedoms. 

Issues Unresolved 

Protection Of Educator’s Intellectual Properties And Distribution Of Materials To Students. 

Most educators in K-12 settings develop materials specific for their personal needs. The intellectual 
properties of those materials are seldom promoted as a topic of discussion at local PTA meetings. However, the 
reality is those materials are indeed copyrightable under the law. Fortunately, many if not most educators tend to 
share those intellectual properties with their peers. Large databases of lesson plans are appearing on the Internet for 
access by prelicensed educators as well as licensed educators. There may be no other profession that so freely gives 
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of its intellectual properties. It would be interesting to see the results if drug conglomerates provided web sites with 
lists of formulas for effective medications. There is no argument under the law that lesson plans are unique and 
tangible expressions of ideas that belong to their creators. If anything, there is a cultural expectation that teachers 
will provide lesson plans and handouts free for all to use. 

Availability Of Quality Materials That Are Free To Educators And Students 

There exists an enormous need for free, quality materials for non-profit educational groups. Many free 
resources are also ones that are not in high demand because they are outdated or not professional prepared. Is there 
a need for a national bureaucracy dedicated to creation of free, useful, quality materials for all grade levels? 
Although the government funds a national printing office for replication of reports and other printed materials 
created by government entities, those materials are not generally geared toward traditional educational settings. Free 
documents geared for professional applications in graduate schools seldom benefit the F‘ grade teacher trying to 
teach about the balance of nature. 

The reluctance of the same federal agencies promoting educational reform to also promote affordable 
educational materials is not unusual. If the idea of educational reform is to truly become effective, then the idea of 
affordable, useful, quality materials must also emerge. Why can’t there be a way to purchase the same textbooks for 
educational purposes so that copyright owners win, publishers win, and most importantly, teachers and students win. 



*V.S Napper & M. Lofgreen. 2003. Catch the chicks before they leave the hen house: Copyright in Teacher 
Education. Presentation. AACE Conference, Albuquerque NM. 

^Ibid. 

^Ibid. 

^Ibid. 

^American Library Association. (1995). Retrieved Sept 25, 2003 from 
http://arl.cni.org/scomm/copvright/uses.html 
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Making a Case for Object-Oriented Instructional Design: 

A Case-based Learning System on Winter Weather Forecasting 



Patrick Parrish 
Bruce Muller 
COMET/UCAR 

Abstract 

The frequently cited promise of learning object strategies for instructional design assumes that finely 
grained instructional content will be frequently borrowed from and used by a wide community to create customized 
instruction. But this promise is still somewhat problematic for many reasons, and it may be some time before we 
know whether scalability benefits will emerge. However, the use of object-oriented design strategies to facilitate 
problem- and case-based learning curricula offers planned (as opposed to speculative) reuse of learning objects, the 
possibility of distributed development efforts, and the potential to create rich learning environments with a variety of 
resources. This paper describes the design of an online Winter Weather Forecasting curriculum that employs 
object-oriented instructional design to offer content-rich, interactive case studies. 

Concerns about Object-Oriented ID 

The use of learning object strategies offers both many promises and many potential problems for the future 
of education and training (Parrish, in press). Much of the touted promise centers on the assumption that efficiencies 
will be created as large repositories of finely grained learning objects are frequently borrowed from and used to 
create custom instruction. This, of course, presupposes that large numbers of content developers will be willing to 
take the trouble to share their content and incorporate the required metadata to make it compatible with these digital 
libraries. In the end, there may be limits to how much borrowing and sharing people are willing to do that may limit 
the use and usefulness of digital libraries. For some, instructional content is also intellectual property to be 
protected. Proper attribution preserved the metadata and embedded in the content may help, but there may remain a 
concern that content quality can be reduced as it is adapted. More importantly, sharing learning objects that can be 
catalogued and stored in large repositories requires work to create useful and detailed metadata. Many potential 
sharers may be put off by the additional effort, or do a less than adequate job. Finally, for many educators, 
instruction is an expressive activity, and the desire to produce original materials may be stronger than the value of 
efficiency. 

The promises attributed to learning object strategies also assume that good, coherent instruction can result 
from using these disparate learning objects to build lessons, that this process will save time, and that a learning 
object approach does not diminish the quality of instruction for the sake of reusability. But there may be constraints 
on a designer’s ability to create meaningful learning experiences using learning objects provided by multiple, 
independent developers. For example, the value of providing a rich context for instructional content is a proposition 
of many theories of learning and instruction, including those focused on learner motivation (Keller, 1983), and those 
positing the situated nature of knowledge (Lave & Wenger, 1991). Because one goal in learning object design is to 
remove context for the sake of reusability, instructional designers are tasked with (a) avoiding an inclination to 
embed context, (b) creating strategies for including context that does not “bleed” into the learning objects 
themselves during their original implementation, and (c) creating an alternative context during reuse. Many of the 
benefits of learning objects assume little instructor intervention, but as the above shows, this is far from the truth. 

A Case-based Approach 

While questions remain about the viability of learning object strategies, some applications show immediate 
and great promise that does not rely on scalability beyond a focused development project. The use of object-oriented 
design strategies to facilitate problem- and case-based learning curricula offers planned (as opposed to speculative) 
reuse of objects, a distributed development effort, and the potential to create rich and varied learning environments 
with a wide range of resources. Similar to object-oriented instructional systems described by previous authors 
(Schank & Cleary, 1995; Bannan-Ritland, Dabbagh, & Murphy, 2002; Orrill, 2002), the COMET® Program is 
employing a learning object approach in creating a case-based learning curriculum for weather forecasters 

The COMET® Program, in association with the Meteorological Service of Canada and the National 
Weather Service (U.S.), is developing a case-based, professional development curriculum for weather forecasters 



280 




titled, Mesoscale Aspects ofWiinter Weather Forecasting. Case-based reasoning (Jonassen & Hernandez-Serrano, 
2002; Kolodner & Guzdial, 2000) has long been recognized in the practice of weather forecasters, who build 
expertise not only by gaining a deeper understanding of meteorological science, but through case experiences where 
situational constraints strongly influence how they apply general and local knowledge. Multiple realistic cases that 
ground scientific knowledge are an essential aspect of initial training for any forecaster, and are equally important in 
professional development that updates scientific knowledge. Case-based learning approaches provide a natural 
environment in which to exploit object-oriented ID strategies by maintaining strong context without excessive 
design effort. 

Each of the planned 15-18 Web-based modules will be based on a case representing a particular winter 
weather forecast challenge. Student performance in each case is supported by short lessons (developed as reusable 
learning objects) that treat knowledge and skills required for good decision making during such cases. Lessons treat 
topics such as the conceptual models and physical processes of winter weather phenomena, cloud microphysics, 
forecasting tools, data analysis, climatology, and public forecast impacts as they relate to winter weather forecasting. 

Each case begins with a Case Challenge that places the learner in the role of a forecaster in a particular 
office, with the goal of forecasting specific aspects of a winter weather event, such as the timing, location, or 
approximate accumulation of snowfall. Resources for the Case Challenge include a rich dataset (including satellite 
and radar imagery, numerical guidance, and other data products provided incrementally as the case unfolds), 
relevant supporting topics (the learning objects described above), and also may include advice from experts or 
experienced forecasters. The Case Challenge is built upon several interactions that replicate typical decisions made 
by forecasters as they develop a forecast, such as the choice of appropriate data products to analyze, judgments 
about the accuracy of numerical model guidance, or the diagnosis of underlying processes revealed in the data. The 
choices the learner makes during the Case Challenge may at times trigger feedback that recommends particular case 
resources that would have helped the learner in making better choices. 

Eollowing the Case Challenge is the Case Analysis, which explores the case in a systematic way and 
provides explanation for what occurred. Like the Case Challenge, this section of the module also calls upon the 
supporting topics (learning objects) to enrich the discussion of the case at hand. Each module culminates with a 
short set of questions testing knowledge of the event’s main characteristics and understanding of key meteorological 
and forecasting concepts underlying the case. 

Benefits to Object-orientation 

The object-oriented approach to creating and providing the instructional resources for the case will facilitate 
several things; 

• distributed development responsibility, allowing content developers at multiple locations to create the 
instructional materials. Since The COMET Program works with national and international experts, as well 
as other training groups, this benefit is large 

• flexible learning options for users, allowing more experienced and knowledgeable forecasters easily to skip 
portions of content with which they are already familiar. Learners may also choose to review by revisiting 
only the collected learning objects or the case summaries 

• increasingly efficient development, through the probable reuse of content in multiple cases (especially later 
in the effort). Many of the topics (for example, the microphysics of snow formation) will be of concern in 
nearly all the potential cases that will be chosen 

In addition, as development proceeds, learning objects can be collected within direct instructional “Eoundation 
Modules” and within content databases that some learners may prefer due to their limited time for on-the-job 
training, and for more rapid reference and performance support. (See Eigure 1 below.) Of course, learning objects 
will also be submitted to appropriate digital libraries such as the Digital Library of Earth Science Education 
(DLESE) ( www.dlese.org ) and the National Science Digital Library (NSDL) ( www.nsdl.org) . 
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Figure 1 . Module and Learning Object Relationships 
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Development Plans 

The first module in this curriculum. Ocean Effect Snow: New England Snow Storm, 14 January 1999, was 
published in March 2003 (http://meted.ucar.edu/norlat/snow/ocean_effect_case/index.htm). By spring of 2004, we 
expect to have 3 additional modules completed and to have gathered feedback from initial users of the products. 
(Feedback so far is limited due to the unseasonable timing of the spring release.) The COMET Program is hopeful 
that this curriculum development effort can serve as a model for similar efforts, and indeed it has already been 
suggested as a potential model by other meteorological training organizations. 

Our second project in the effort is being done in collaboration with the UK Met Office College and the 
European meteorological training consortium, EuMetCAL. The Met Office is adopting these case-based and object- 
oriented strategies in our joint effort to create training on how to forecast Polar Low events. The project is being 
carried out by a distributed project team, and will include both a North American and European case challenge, 
accompanied by a set of shared and co-developed supporting topics as learning objects. Even though the final 
products used in the U.S. and European will be in different formats (Flash-based in the U.S. and XML/HTML-based 
in Europe), the same content drafts will form the basis for both, diverging only in the final phases of development. 

The object orientation of our design is what makes such collaboration and division of labor possible. This 
strategy will leverage our resources to better serve our broad community of users, which also includes military 
forecasters, university departments of meteorology, and other international weather services. Collaborative projects 
such as the one with the UK Met Office and EuMetCal will increase the development capacity of all collaborating 
groups, and will also broaden to the repertoire of cases and techniques for learners. 
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Dropping your anchors: Implications for learning anchors in traditional 

classrooms 



Drew Polly 
University of Georgia 

The increased access to technologies in schools has opened avenues to explore non-traditional styles of teaching 
and learning. Educational theorists and researchers have long been calling for learner-centered instruction that 
situates learners in activities that allow them to explore concepts and construct understanding. However, as 
constructivist theorists and researchers continue to show the benefits of situating learning in meaningful tasks, 
many barriers still prevent the use of technology-enhanced authentic activities in classrooms (Shaw, 2003; 

ISTE, 2002). This paper aims to analyze the underlying theories of authentic learning and propose methods to 
support classroom teachers with the design, implementation and assessment of authentic activities. 



Technology In Schools 

As computers and other technological devices arrive in schools they continue to be an underused resource 
to anchor learning in authentic learning environments. The latest reports from Market Data Retrieval (2002) cite that 
the ratio between students and instructional computers is an all-time low of 3.8 to 1. Eurther, the report states that 
there are 4.9 students to each Internet-connected computer. Becker and Ravitz (2001) surveyed secondary teachers 
and found that less than 25 percent of them were using computers on a weekly basis to enhance instruction. In most 
cases the technology is used in drill and practice settings, rather than promoting process or higher-order thinking 
skills (Becker, 2001; Shaw, 2003). 

Technology’s potency to anchor learning in authentic environments has been long running. Bransford et al. 
(1990) contended that video discs and other technologies can be employed to anchor instruction in authentic tasks. 
Hannafin (1992) advocated using electronic tools and resources in open-ended learning environments (OELEs) to 
allow students to investigate real-life problems. Bransford and his colleagues in the Cognition and Technology 
Group at Vanderbilt (CTGV) (1992) provided a significant bridge from theory to practice with the Jasper series, a 
set of videodiscs that anchors mathematical problem solving in authentic narratives. Students watched the videodisc, 
then identified and solve problems that were embedded in the story. The Jasper series was the first of many 
technology-rich activities that attempt to situate learning in authentic contexts. CTGV’s endeavor to promote 
authenticity via technology has been promoted by other researchers (see Griesser, 2001; Herrington & Oliver, 1999; 
Shyu, 2000). 

Those studies continue to support the effectiveness of student learning in technology-enhanced authentic 
learning environments (CTGV, 1992; Grieser, 2001; Shyu, 2000;). Such an onslaught of research, however, begs the 
questions: what theories are underpinning the notion of authentic activities, and if authentic learning is such an 
effective method of instruction, why isn’t everyone using it? 

Theories of Authenticity 

The term authenticity has become commonplace in education, yet its definition remains obscure and 
undefined (Barab, Squire, & Dueber, 2000). Authentic learning environments, traditionally, describe structured 
activities that have meaningful context (Radinsky et al., 1998). The term “meaningful context”, however, is not the 
same for all people. Eor example, while investigating the migrating patterns of buffalo may be authentic for children 
in the midwest, that activity lacks meaning for students who live in a fishing village on the coast of the Atlantic 
Ocean. Authenticity lies in the eyes of the beholder, since an experience is authentic if and only if the learner can 
derive meaning from the experience (Barab & Duffy, 2000). 

Authentic learning is akin to Dewey’s (1910/1978)’s notion of learning through experience, as well as 
Vygotsky’s theory of social learning (1987). Dewey cited the importance of learning through various experiences in 
life that the learner can participate in. Vygotsky (1987), meanwhile, contends that schooling should introduce 
learners into various communities of practitioners. In science education courses, Vygotsky would advocate 
participation in experiences of scientific investigation and inquiry that resemble the activities that practitioners 
engage in. 
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When schooling is based on authentic tasks, students become more responsibility for their learning, and 
actively participate in the collection and analysis of information, as they seek ways to solve specific problems. Most 
activities bridge many disciplines, bridging numerous content areas, thus enabling them to master content more 
effectively (Barab and Linda, 2000; Brown, Collins and Duguid, 1989). 

Paradoxically, education primarily remains the home of abstract and decontextualized instruction. Brown et 
al. (1989) concluded that school activities lack value in the real-world, since formal schooling has very little transfer 
to real-world practice and they contended (p. 36) that “success within school often has little bearing on performance 
elsewhere.” Recent studies (Becker, 2001; Greisser, 2001; Shaw, 2003; Shyu, 2000) have confirmed that although 
instruction has become more engaging for learners, classrooms still remain a place where students engage primarily 
in drill and practice activities, providing minimal transfer to real-world settings. In order to transfer to real-world 
contexts to occur, instruction should be situated in contexts that are as life-like as possible. 

Components of Authentic Learning Environments 



Four primary components emerge from the body of literature concerning learning that is situated in 
authentic environments. They are cited in the table below: 



Component 


Summarv 


Literature 


Meaningful 

Tasks 


Learners should participate in tasks that they 
can derive meaning and purpose from. Tasks 
should maintain the interest of students and 
encourage the transfer of knowledge from the 
learning environment to other situations. 


Bransford et al. (1990), Brown et. 
al (1989), Dodge (2003), Moore et. 
al (1994), Scardamalia & Bereiter 
(1996), Shaw (2003). 


Open-Ended 

Environment 


Activities should be open-ended and be able 
to be solved in multiple ways. The activity 
should emphasize the problem solving 
process, rather than encouraging the student 
to simply find the correct answer. 


Barab (2000), Bransford et al. 
(1990), Greeno (1998), Moore et. 
al(1994) 


Scaffolds to 
support learning 


Scaffolds should be available to the learner 
during the learning process. Examples could 
include a teacher facilitating the problem 
solving process, the modeling of an expert, 
or just-in-time resources. 


Herrington and Oliver (1999), 
Oliver and Hannafin (2001), Shaw 
(2003), Vygotsky (1987). 


Opportunities to 
reflect 


Reflective thinking should encourage 
learners to think about the problem-solving 
process and reason through other points of 
view. Reflection increases the likelihood of 
knowledge transfer to similar situations. 


Brush (2003), Bruning (1999), 
Dodge (2003), Greisser (2001), 
Herrington and Oliver (1999). 



Examples of Authentic Learning Environments 

While the idea of authentic learning is promising, teachers are unlikely to employ such activities without 
concrete examples. Below are two examples, one for seventh grade mathematics and one for a high school history 
class. 

Seventh grade mathematics: You receive a credit card in the mail. The credit card has a minimum payment of $25 a 
month and a 10% annual interest rate. You want to buy a $600 entertainment system. If you earn $50 a month 
babysitting, examine your options of paying off the bill? Which option is the cheapest? If you only made $50 a 
month babysitting, how would the price you paid for the entertainment system change? 

First, the task is meaningful, as students must form an understanding of credit, debt and interest. The 
activity is open-ended with many ways to solve this problem. Activities such as this lend themselves to employing a 
spreadsheet to provide a visual representation of the data, as well as explore various variables that affect the 
problem, such as how much of the debt is paid each month. The teacher can serve as a scaffold, providing support 
on identifying the problem, choosing an approach to the problem and key questions to focus on throughout the 
activity. The use of a spreadsheet or table can also scaffold learning, by providing a display of data that the learner 
can manipulate in order to find a solution. This activity also has the potential to be reflective as well. A spreadsheet 
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or calculator will allow students to collect data for various situations, such as 1 ) paying off the debt as soon as 
possible, 2) paying slightly more than the minimum payment or 3) paying only the minimum payment. Comparing 
the various scenarios enables the learner to choose which option is the best, and allows generalizations to drawn 
from that information. 

While authentic learning activities are pretty realistic in mathematics, other life-like situations can be 
created for other subjects. 

High School government: You are the campaign manager for Douglas Dooright, a candidate 
for U.S. President. The Presidential Election will be on November 6^^. Douglas will by flying 
around the country making his final bid for votes from October 29-November 5. If he can only 
make three speeches a day and has only $3,000 to spend on airfare, what will his itemized 
travel budget look like? Explain why you chose to travel to each location. 

The task described is life-like and meaningful, as the learner takes on the role of a campaign manager, 
planning the speech schedule for a presidential candidate. This activity builds on skills such as scheduling, money 
management, and comparing travel fares, which are all meaningful tasks to high school age students. This activity is 
extremely open-ended, allowing students to decide when traveling occurs, where they will travel and which airfare 
to use. Once again, the teacher has the potential to serve as a scaffold, as students interpret the problem, choose their 
approach, and make decisions during their activity. Technology, mainly the World Wide Web, also scaffolds 
learning by providing just-in-time information on electoral votes, key locations and travel fares whenever the 
students need it. While, the immediate access to information helps the student in the activity, it has the potential to 
be a hindrance. Oliver and Hannafin (1999) cited dilemmas with students with managing information in OELE’s as 
learners became inundated with data. This problem can be lessened if the learner participates in reflection and 
focuses on the problem solving process. Each student is going to have a different travel schedule, since there are 
numerous cities and numerous airline flights each day. The learning that occurs is not simply how to design a travel 
itinerary, but how to engage in data-based decision making and support their choices in a logical manner. 

Technology’s Role 

It is obvious that technology serves as a potent resource in authentic activities. In the first example, the use 
of a spreadsheet allows students to visual represent their investigation in a chart. By using a spreadsheet, students 
can set up the chart using formulas and investigate scenarios based on how much they pay each month. A calculator 
could also be used instead of a spreadsheet. But, while a calculator facilitates the calculation of interest and 
payment, it does not provide a dynamic representation in the form of a chart like a spreadsheet. Without the use of a 
spreadsheet the activity could be completed, but becomes more focused on performing calculations, rather than 
representing a problem and making data-based decisions. 

In the second activity, the Internet and airfare websites such as www.orbitz.com students have access to 
airfares quickly. A spreadsheet could then be used to store their records and keep a running tally of their travel 
budget. The process of completing this activity resembles a traditional webquest (Dodge, 2003), in which students 
use the Internet to find information and apply it to solve a problem. 

In both activities, access to technology is essential. Students must not only be able to get to the technology, 
but also must have the requisite knowledge to complete the activities (NRC, 2002). In both activities, the requisite 
knowledge includes being able to set up and manipulate a spreadsheet. The second activity also requires the student 
to know how to search the Internet for information. 

Technology skills are now a part of curricula nationwide, as students learn word processing skills, how to work with 
spreadsheets, and how to create products such as presentations and webpages (Market Data Retrieval, 2002; Sargent, 
2003). Employing technology-rich authentic activities supports current research (NRC, 2002; Market Data Retrieval, 
2002) that students should learn these technologies by using them in life-like contexts. 
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Put the Learner First: Cognitive Theories in Instructional Design 

Drew Polly 
University of Georgia 

Introduction 

Instructional design models encompass a wide gamut of theories and applications in education and training 
environments. Gustafson and Branch (2002) described a taxonomy with three specific purposes for the development 
of instruction. The three categories encompass 1) development of instruction for individual classrooms, 2) product 
development for implementation for users other than the developers and 3) more complex instructional systems 
instructed directed at an organization’s problems. 

Classroom instructional design models are of great interest to those who see their role as a teacher to students. Users 
of these models include K-12 teachers, university faculty and some training programs in business and industry 
(Gustafson and Branch, 2002). The Morrison, Ross and Kemp model serves as a framework to develop effective 
classroom instruction for instructional designers. The model originated with Kemp’s original design in 1971 and 
has evolved in the past 30 years into its present form. 

Conceptual Basis 

Morrison, Ross and Kemp’s model is holistic in nature. It is based on the concepts that instruction should 
be learner-centered, engaging and designed specifically to meet the characteristics and needs of the learner. Further, 
the model employs the concept that the conditions of the learning environment are significant to the effectiveness of 
instruction. 

As the model evolved, Morrison became the lead author, but has maintained the same learner-centered 
concepts that Kemp used in its original development (Gustafson and Branch, 2002). Morrison has significant 
background in training and sees the benefits of cognitive based approaches. Morrison (personal communication, 
January 30, 2003) explained, “both Steve Ross and I were trained with a heavy emphasis on Bruner’s work, which 
lent itself to the orientation of this model. In addition, my experience in business and industry convinced me that 
content-oriented models were not very effective.” 

Theoretical 

Morrison, Ross and Kemp adhere to the belief that instruction and instructional development should be 
learner-focused. Bruner’s cognitive-based theories can be attributed to the foundation and evolution of this model. 
Bruner (1973) posits that instruction should be concerned with three main issues. First, instruction should be based 
on the experiences and contexts that make the student willing and able to learn, known as readiness. Second, 
instruction should be structured so that the learner can effectively grasp the content. Finally, instruction should be 
designed to facilitate the extrapolation and application of knowledge, allowing the learner to construct their own 
meaning. 

Morrison, Ross and Kemp (2001, p. 4) advocate asking a series of questions during the development 
process. As illustrated in the figure below; four of the questions coincide with Bruner’s theories of cognitive-based 
instruction. 



Figure 1: 



Morrison, Ross and Kemp’s essential questions 


Bruner’s theories 


1) What level of readiness do individual students need for 
accomplishing the objectives? 


Instruction should account for the 
learner’s experiences and readiness. 


2) What instructional strategies are most appropriate in terms 
of objectives and learner characteristics? 


3) What media or other resources are most suitable for 
instruction? 


Instruction should be structured so that 
the learner can effectively grasp the 
concept. 


4) What support is needed for successful learning? 



The other two questions address the assessment of instruction and the development process: 5) How is achievement 
of objectives determined? 6) What revisions are necessary if a tryout of a program does not match expectations? 
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The theories of cognitive psychologists advocate instruction to be learner-centered and developed with the 
learner’s characteristics in mind. The Morrison, Ross and Kemp model applies cognitive concepts and theories to 
the development of instruction by keeping the process focused on the characteristics and needs of the learners. 

Operational 

The Morrison, Ross and Kemp model expresses their views that Instructional Development is a cyclical, 
systematic process with revision as a continuous activity (Gustafson and Branch, 2002). The developer of 
instruction, according to Morrison, Ross and Kemp, should attend to nine main elements. These elements are 
interdependent and can be completed as appropriate (Morrison, personal communication, January 30, 2003; 
Gustafson and Branch, 2002). 

Morrison posits that developers starting afresh should first define the instructional problem. While some 
developers contend that development can begin with the task analysis, Morrison (2003) claims that, “a task analysis 
can not be done without knowing focus of the instruction (i.e., a goal analysis). The problem must be defined before 
other tasks, if the designer is starting from scratch.” 

The current model varies slightly from the operational nature of Kemp’s original model. In 1971, Kemp 
established a linear sequence for developing effective instruction. The model changed into an oval pattern (see 
Appendix A), which takes away any notion of a linear sequence (Kemp, Morrison and Ross, 2001). Kemp, 

Morrison and Ross (2001) wanted to symbolize the non-linear model of instructional development, allowing 
individuals have the flexibility to proceed through the process in their own preferred way. 

The one aspect that has been consistent throughout the evolution of this model has been the significance of 
revision in the process of instructional development. Kemp’s original model called for formative revision to occur 
after each step in the process. In the current model, formative evaluation and revision are integrated into every step 
of the instructional development process. In addition to formative evaluation, the current model calls for summative 
evaluation at the end of the development process. Further, confirmative evaluation is recommended to occur months 
after completion to evaluate both the instruction and the materials that were developed. 

Analysis 

In comparison to the ADDIE model, Morrison, Ross and Kemp’s model has a multitude of analytical 
components. Analysis occurs during the first three elements of the model. In order for the developer to define the 
instructional problem, discover the characteristics of the learner, and conduct a task analysis all require the 
developer to use a similar process to the Analysis stage of the ADDIE model. 

Appendix B shows McGriff’s realignment of Morrison, Ross and Kemp’s model to fit into the components 
of the ADDIE model. The model’s learner-centered nature requires the developer to thoroughly understand the 
learner before instruction is developed. The following analyses are incorporated into the model; needs analysis, 
learner analysis, analysis of instructional strategies, task analysis, goal analysis and problem analysis (Kemp, 
Morrison and Ross, 2001). In the model, the analysis component is the most thorough (Morrison, personal 
communication, 2003). 

Design 

Two of the nine elements in Morrison, Ross and Kemp’s model coincide with the design process of the 
ADDIE model. Both the instructional objectives and the sequence of content are designed in Morrison, Ross and 
Kemp’s model. Instructional objectives should be written in terms of the learner and cover the three primary 
domains (Kemp, Morrison and Ross, 2001). 

The focus of instructional design should be the cognitive domain, which includes objectives dealing with 
information, knowledge or other intellectual aspects of learning. Coinciding with Robert Gagne’s work, Morrison, 
Ross and Kemp contend that instruction should extend to the higher levels of the cognitive domain, such as the more 
advanced stages of Bloom’s taxonomy. Instruction should also account for the psychomotor domain, which 
includes skills associated with the use of skeletal muscles. 

Einally, instructional objectives should be based include the affective domain, an area encompassing attitudes, 
values and emotions. 

Development 

While one would expect an instructional development model to be heavy-laden with phases of 
development, Morrison, Ross and Kemp’s model is contrary to that. The model incorporates steps for the 
development of instructional strategies and the design of the message. The learner-centered nature of the model 



290 




establishes a standard for instructional strategies to include engaging activities that allow the learner to take 
responsibility in constructing their own knowledge. 

The design of the message must account for the characteristics of the learner and incorporate media that 
effectively present the content during instruction. Morrison, Ross and Kemp’s use of a cognitive-based learner- 
centered model requires the media to be effectively integrated in the curriculum in such a way to make learning an 
active, constructive process. 

Implementation 

The instructional delivery and the use of evaluation instruments fall into the Implementation phase of 
Morrison, Ross and Kemp’s model. The essential point in the implementation phase of the model is the need for 
revision. While formative evaluation has occurred during each part of the instructional development process, the 
largest impact of formative evaluation occurs iust before the implementation of instruction (Morrison, Ross and 
Kemp, 2001). 

The implementation of evaluation is also a critical part of the model. However, Morrison recognizes 
problems with this element of the instructional design model for industry training. Morrison (personal 
communication, January 30, 2003) contends that, “testing is not always welcome or well-suited for the industry 
environment. Often times, it is left out of the training altogether.’’ 

Evaluation 

As mentioned previously, Morrison, Ross and Kemp fully endorse the constant use of formative 
evaluations to monitor the process of developing effective instruction. Three means of evaluation are used in the 
model. Formative evaluation occurs throughout the instructional development process, and provides the designer 
with easy accessibility to making changes as they are needed (Morrison, Ross and Kemp, 2001). 

Summative evaluations occur at the end of the instructional development process and attempt to evaluate 
the development of instruction on a larger scale. Further, the model uses confirmative evaluation a long-term after 
the implementation of instruction. Confirmative evaluation analyzes both the instruction and the materials that were 
used during instruction (Morrison, Ross and Kemp, 2001). 

Orientation 

This holistic model is comprehensive and has both prescriptive and descriptive orientations. Edmonds, 
Branch and Mukherjee (1994) explained that “descriptive models outline a given learning environment and imply 
how the variables of interest will be affected in the environment based on practical experiences of instruction and 
learning (p.60).” This model is descriptive since it bases every factor on the practical experiences of learning and the 
specific needs of the learner. Every analytical component of the model lends itself to its descriptive nature. 

Paradoxically, prescriptive models focus on how learning environments can be manipulated in order to 
bring about desired outcomes (Edmonds et ah, 1994). The elements of sequencing the content, developing 
instructional strategies, designing the message and developing instructional and evaluation materials contribute to 
the model’s prescriptive attributes. 

Knowledge Structure 

Morrison, Ross and Kemp’s model resembles traditional declarative models. By developing instruction 
from the vantage point of the learner the learner accounts for each learner’s characteristics and learning needs 
(Gustafson and Branch, 2002). Their first essential question is “what level of readiness do individual students need 
for accomplishing the objectives.” The essential questions and elements of the model frame a developed product that 
is much more focused on learner-centered instruction that is catered to the individual needs of the students. 

Expertise Level 

Morrison, Ross and Kemp’s model can be employed successfully by the most novice of designers. The 
nine elements are clearly laid out and establish a framework that allows designers to gain information about the 
instructional problem and the learner prior to designing instruction. The model’s flexibility is a strength, with its 
interdependent elements that permit users to operate where modifications are needed (Gustafson and Branch, 2002). 

The pre-established essential questions and algorithmic steps and procedures are characteristics of models 
that are appropriate for novice instructional designers (Edmunds et al, 1994). As a novice designer in the business 
and industry sector, the model systematically walks the user through each individual step. The narrative (in 
Morrison, Ross and Kemp, 2001) provides the theoretical framework and implications for use, but is not necessary 
for an instructor to design effective instructional units. 
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Structure 

A designer starting anew would begin the design process by identifying the instructional problem prior to 
completing any other steps. Due to the model's dependence on the instructional problem and the characteristics of 
the learner, the model can be categorized as a hard-systems instructional design model. 

Gustafson and Branch (2002) describe hard systems as models aiming for the efficient attainment of goals. 
Hard systems advocate goal seeking as an appropriate model of human behavior. Morrison, Ross and Kemp's model 
focuses on helping the learner to master the established objectives, coinciding with the notion of attaining goals in 
instruction. 

Context 

Gustafson and Branch (2002) classify the Morrison, Ross and Kemp model as a classroom-oriented model 
that is user-friendly to both K-12, business and industry instructors. Morrison (personal communication, January 30, 
2003) claims, “the focus is mainly on training for business and industry, but is unique since most training for 
business is content-centered, rather than being learner-centered.” Morrison, Ross and Kemp (2001) explain in 
Chapter 1 that K-12 educators cannot afford the time to thoroughly engage in the instructional design process, which 
led them to framing this model towards a business context. 

Level 

Morrison, Ross and Kemp's model provides a framework for designing a lesson or unit's worth of training. 
The model's thorough analysis of the learner and its learner-centered nature prevent this model from being used 
efficiently on scales, such as institutional, modular or curriculum. Unit-level instructional design determines the 
specific topics and tasks that will be learned (Gustafson and Branch, 2002). Morrison, Ross and Kemp's emphasis 
on the events during instruction also allows a designer to plan a single lesson using the model. 

Scholarly Opinion 

Morrison, Ross and Kemp made a significant contribution to the field of instructional design with the 
development of their learner-centered model. Breaking away from the traditional content-based models, this 
framework employs a cognitive approach that designs instruction based on the characteristics and needs of the 
learner. 

This model’s underlying concepts and theories are concise and effectively represented in its operational 
nature. The essential questions and elements of the model clearly employ Bruner’s theories of instruction and the 
concepts of creating learner-centered instruction based on the information collected during the analysis stage. 

Morrison, Ross and Kemp’s model is heavy in its analytical components. The numerous types of analysis 
establish a strong background to help the designer develop instruction. But, are they all needed? “When I ask most 
students which step could be removed, most students answer the learner analysis,” Morrison wrote. “I do not think 
you can do away with this step, but it is probably the step receiving the least amount of attention in most cases.” 

For a designer starting from the beginning of the design process, a thorough analysis is essential to 
effectively design instruction. However, the strength of this model is the interdependent elements, allowing 
designers to work on whichever elements are needed in any particular order. Hence, designers that have a strong 
sense of the learners and already have their instructional objectives defined, can begin the design process at a stage 
somewhere else in the design process. The oval shape of the model conveys the non-linear focus of instructional 
design. 



Conclusion 

Morrison, Ross and Kemp’s model provides a framework for developing instruction based on the learner’s 
instructional needs. The learner-centered concepts and cognitive theories of learning are clearly conveyed in the 
essential questions and elements of the model. Heavy-laden with analytical components and opportunities for 
constant revision, the model encompasses all aspects of instructional design and provides a sound backdrop for 
providing effective instruction. 
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Abstract 

Headings are commonly-used verbal signals that divide instructional text into smaller sections and prime 
the reader to the main ideas discussed in those sections. Writers of instructional text may incorporate headings into 
their work without realizing that it impacts the reader’s ability to remember and process the information. In this 
review of literature, the effects of headings and other verbal signals on learning are explored. Of the studies 
reviewed, 15 are discussed and synthesized to form practical guidelines for instructional designers and educators. 
Among the findings, headings have been shown to promote recall of main ideas, knowledge of organizational 
structure, and problem-solving. However, even as signaling makes key topics more memorable, it can also inhibit 
recall and comprehension of subordinate information that is not signaled. These and other findings are presented 
and discussed. 



Introduction 

Headings represent one member of a large family of instructional variables called signals. Verbal signals 
include titles, introductions, overviews, headings, subheadings, transition statements, and summaries. Non-verbal 
typographical signals include bolding, italics, underlining, spacing, all-caps, font styles, font sizes, color text, and 
color highlighting. In instructional settings, verbal and non-verbal signals help learners navigate bodies of 
instructional text by providing structure, sequence, organization, and flow. Signals are used in the expectation that 
they make the relevant topics stand out in importance and thereby more memorable (Mautone & Mayer, 2001). 

Headings, like other verbal signals, are “non-content words (that) make the semantic content and structure 
of expository paragraphs more explicit” (Mautone & Mayer, 2001; p. 377; See also Meyer, 1975). Headings divide 
the instructional text into smaller chunks, presumably making the information easier to encode. Readers can make 
use of headings as priming aids before reading (to help them skim the passage), as encoding aids during reading (to 
orient themselves to the text that follows), as retrieval aids after reading (to cue retrieval of a particular concept), 
and as search aids (to assist in locating specific information). Signals research has typically focused on the encoding 
effects of headings (Hartley & Trueman, 1985). 

The encoding effects of headings have been shown to vary by a number of factors (Hartley & Trueman, 
1985; Marcinkiewicz & Clariana, 1997). For example, Meyer, Brandt, and Bluth (1980) found that headings in text 
enhanced recall for poor readers but not for good readers. Conversely, Hartley and Trueman (1985) found that 
heading statements aided high-ability students most and low-ability students least. Such differences are common in 
headings research and are, in part, the result of differences in the treatment and assessment materials. However, 
these inconsistencies are also indicative of the subtle yet complex relationship that exists between the internal 
memory processes of encoding and retrieval and the perception of signals. The following is a review of the most 
pertinent studies. 



Review of Literature 

In the absence of headings and other signaling devices, which highlight the intended organizational 
structure of the reading passages, the reader utilizes a linear approach to reading the text. The ideas are stored in 
memory in the form of a serial list. The learner’s processing of instructional content and his/her ability to recall or 
summarize key concepts is then highly dependent upon the content’s serial positioning (Loman & Mayer, 1983), the 
textual elaboration of key points (Torch, Torch, & Inman, 1993), and the learner’s prior knowledge of the subject 
matter (Torch & Torch, 1996b). These factors are “secondary indicators of importance of text topics” (Torch, 
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Lorch, Ritchey, McGovern, & Coleman, 2001; p. 172). The presence of signals, however, can mitigate these 
factors, by focusing the learner’s attention on the organizational structure of the content. Signaling devices enable 
the reader to use a structure strategy in which ideas are associated with higher-level topics within a broader 
organizational structure (Sanchez, Lorch, & Lorch, 2000). This should make it easier for readers to access ideas 
even when those ideas are not positioned at the beginning or end of the text. Signaling should improve recall of 
those ideas. 

Despite this common sense hypothesis, Lorch and Lorch (1996b) note that some prior studies have failed to 
find significant signaling effects on total recall, i.e., the overall number of idea units recalled (Brooks, Dansereau, 
Spurlin, & Holley, 1983; Lorch et ak, 1993; Meyer, 1975; Meyer & Rice, 1982). Even so, headings and other 
signaling devices have been shown to improve recall under certain conditions, for certain types of information, and 
for certain learners. 

Do signaling effects vary by signal type? 

Ejfects of headings, summaries, outlines, and underlined text on comprehension. In a study involving 432 Air Force 
basic trainees, Christensen and Stordahl (1955) examined the effects of signals on comprehension by presenting 
students with an outline, a summary, underlining, and/or paragraph headings. Using a factorial design, the 
researchers examined 35 different possible combinations of signals in comparison with a no signals control group (n 
= 12 per group). The pretest and posttests used consisted of multiple-choice test items. No significant interactions 
were uncovered for either pretest or posttest scores. Neither headings alone nor in combination with other signals 
had any appreciable impact on comprehension. 

Ejfects of headings, previews, and connectors on topic recognition. In an effort to isolate the effects of headings, 
previews, and logical connectors on the retrieval of both inferential and detailed information, Spyrikakis and Standal 
(1987) examined all three signal types individually and in tandem with each other. The study involved 300 college 
students who each read one of four passages (topics were nitrates, corrosion, algae, and biomedical) of varying 
structure, depth, and length (ranging from 362 to 766 words). Each also completed one of four 10-item multiple 
choice tests measuring the ability to recognize both super-ordinate topics and subordinate information in the 
passages. The use of headings was found to be significantly beneficial with only one of the four passages 
(biomedical) and then only with regards to enhancing super-ordinate/inference comprehension, F(l, 64) = 4.19, p < 
.05 (p.291). The biomedical passage was the longest (600-766 words), the richest in terms of the number of topics 
(42 versus 36 for the next highest), and the most difficult in terms of grade level. This implies that headings are 
most useful for stimulating recognition of key points (super-ordinate topics) when the content is long, difficult, and 
complex. Nevertheless, Spyrikakis and Standal (1987) suggest that in this study, “the results are suspect due to a 
ceiling effect” (p. 291). 

Ejfects of headings and outlines on free recall. Krug, George, Hannon, and Glover (1989) conducted a series of 
experiments testing the impact of presenting outlines (prefacing text) and headings (interspersed within the text) at 
encoding on students’ free recall of prose. In the first experiment, involving 62 undergraduate Education majors and 
a 600-word essay containing 48 idea units, Krug et al. found that both headings and outlines improved recall equally 
well and that a combined headings/outline approach was the most effective, F(3, 58) = 10.44, p < .01. In their 
second experiment, Krug and his colleagues asked 56 students to read a 5500-word textbook chapter - in an effort to 
confirm the results of the first experiment in a more realistic classroom context - and again, headings and outlines 
improved recall equally well. The combined approach of headings with outlines was the most effective. 

Ejfects of headings, overviews, and summaries on free recall. In a study designed in part to isolate the effects of 
different signaling devices, Lorch and Lorch (1995) tested the individual effects of headings, overviews, and 
summaries on free recall of main and subordinate topics. College students (n = 274) were given a 1,750-word 
passage that varied in terms of the presence or absence of the three signaling devices, with 3 1 of those participants 
receiving the headings passage. All three signaling devices improved recall of the 12 main topics (topic access), 

F(l, 176) = 37.39, p < .05, and conditional recall of subordinate topics, F(l, 176) = 12.29, p < .05 (p. 541). All three 
signals enhanced recall equally well. As a result, the results of all three signaling conditions were combined and 
analyzed with relation to the presence or absence of recall cues. Consistent with the encoding specificity hypothesis 
(Tulving & Thomson, 1973), Lorch and Lorch found that the presence of signals at encoding (in the passage) and 
recall cues (during the free recall task) significantly enhanced recall of both main topics and subordinate 
information. Interestingly, in the absence of recall cues, signaling only during encoding depressed recall scores, 
although these effects did not reach significance. The encoding specificity hypothesis predicts that context cues 
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presented at encoding are most effective only when they are presented again at the time of retrieval. The findings in 
the Torch and Torch (1995) study suggest that signals operate in a similar manner as other context cues, enhancing 
recall the most when present at both encoding and retrieval. 

Do signaling effects vary by ability level? 

Signaling ejfects on cued recall by ability level. In a study measuring the impact of titles and paragraph headings on 
recall of prose text, Hartley, Kenely, Owen, and Trueman (1980) found that using titles and headings improved 
recall on 8 short-answer questions. The 175 second-graders who participated were blocked by ability level; asked to 
read a 400-word passage with (A) no title or headings, (B) with title but no headings, (C) with title and underlined 
passage headings written in the form of statements, or (D) with title and underlined passage headings written in the 
form of questions. Although they did not find significant recall differences between (C) headings worded as 
statements and (D) those worded as questions. Hartley et al. did find that conditions C and D combined - regardless 
of format - prompted significantly higher recall than conditions A and B combined, F{3>, 171) = 4.88, p <.01. This 
suggests that headings, regardless of the format, improve the recall of prose. A significant interaction was also 
found between the use of headings and ability level. Tow ability students benefited more from headings than high 
ability students, F{b, 151) = 2.47, p < .05 (p. 306). 

Do signaling effects vary by item difficulty, familiarity, or prior knowledge? 

Headings ejfects on retention and prior knowledge. Wilhite (1988) found a significant interaction between prior 
knowledge (1988) and retention. The study involved 1 16 college students who were asked to read an 8-section, 
1,760- word passage with or without headings and answer 24 multiple-choice questions (8 low-level topic pretest 
items, 8 low-level topic posttest items, and 8 high-level posttest items). When means were adjusted with the prior 
knowledge covariate (pretest score), headings were shown to enhance retention of main-idea topics, F(l, 111) = 

7.51, p = .007, but not for low-level detail (p. 220). Also, this effect only applied consistently to students with high 
prior knowledge, and not for the low prior knowledge ones. From these results, Wilhite concluded that: 

. . .part of the beneficial effect of headings derives from their tendency to activate relevant schemas in the 
reader during the encoding of the passage material. . .(and not) by promoting the interrelating of concepts (or by 
influencing) retrieval by serving as cues at the time of test for subjects with low preexisting knowledge, (p. 223) 

In other words, with recognition tasks (e.g., multiple-choice tests), learners with higher prior knowledge are 
able to make use of headings because the headings activate already-existing schemas. This, in turn, helps these 
students remember the main ideas of a passage, but does not necessarily help them remember more of the 
subordinate information. Tearners with low prior knowledge do not benefit from the headings. 

Headings effects on multiple-choice test performance by item difficulty. In a study that examined the retrieval 
effects of headings in a 43-item multiple-choice test to demarcate topic sections, Marcinkiewicz and Clariana (1997) 
found that headings significantly improved test performance for low {es = 0.92) and medium {es = 0.44) difficulty 
items but had no effect for high difficulty items {es = 0.02). This study involved 143 mostly-male, adult workers. 

No expository prose passages or lists were used as lesson material. The results of this study varied from a similar 
one (Townsend, Moore, Tuck, & Wilton, 1990) in which no significant differences for headings were found. 
Marcinkiewicz and Clariana (1997) attribute the difference primarily to the fact that they accounted for item 
difficulty level and the previous study had not. The results of this study are consistent with Wilhite (1988) in that 
easier test items (high prior knowledge) were influenced by signaling effects while difficult ones (low prior 
knowledge) were not. 

Headings ejfects on free recall by familiarity. In two experiments examining the impact of headings on recall of 
prose. Torch and Torch (1996a) had 80 Introduction to Psychology undergraduates read a 2,400- word passage 
containing 12 topics and then complete a free recall task. Half of the topics were rated as familiar and half as 
unfamiliar. Although familiar topics were recalled more often than unfamiliar ones, the presence of headings in the 
passage improved overall recall of unfamiliar topics, F{\, 60) = 6.02, p < .05, but had no effect on familiar topics (p. 
268). Consistent with other signaling research (Torch & Torch, 1995, 1996b), headings significantly improved 
recall of unfamiliar main topics, /'’(I, 60) = 4.30, p < .05, but had a slightly negative effect on recall of unfamiliar 
subordinate topics (p. 269). 

Headings ejfects on text summarization by familiarity. In a second experiment involving 99 undergraduates and a 
3,700-word reading passage containing 20 topics. Torch and Torch (1996a) found that headings improved recall on 
a summarization task for both familiar, f(76) = 2.96, p < .05, and unfamiliar topics, f(76) = 3.68, p < .05 (p. 272). 
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Also, headings enhanced recall of topics that were only briefly discussed in the text, t(16) = 4.99, p < .05, but had 
little influence on topics that were discussed at length. 

Do signaling effects vary by text length and complexity? 

Signaling effects on recall of main and subordinate topics. Lorch et al. (1993) conducted two experiments testing 
the effects of signaling (i.e., headings, blank lines inserted between sections, overviews, and summaries) on recall of 
main and subordinate topics. In experiment one, subjects were 203 Introduction to Psychology students. Each was 
asked to read one of eight versions of a reading passage, which varied based on length (3,600 words with 24 idea 
units or 2,400 words with 12 idea units), complexity (2-level vs. 3-level topic structure), and signaling (all present 
vs. all absent). Signaling improved recall, F(l, 194) = 67.00, p < .05; signaling had greater impact on shorter text 
(2,400 words) than longer text (3,600 words), F(l, 194) = 5.51, p < .05; and signaling improved recall 
proportionately more for text with a simpler topic structure (2 levels) than text with a more complex organization (3 
levels), F(l, 194) = 1 12.73, p < .05 (p. 283). To summarize, signals were effective encoding aids and were more 
effective when the text was short and/or simple (2 levels and 2,400 words) than when it was long and/or complex (3 
levels and 3,600 words). 

Signaling effects on recall of short and long discussion topics. In experiment two (Torch et al., 1993), 82 subjects 
were asked to read a 2,000-word passage that varied based on signaling (all present or all absent) and the counter- 
balanced mixture of short and long discussions of topics. Signaling improved recall of subordinate topics that were 
embedded in shorter discussions of a topic, F(l, 78) = 35.53, p < .05 (p. 285). In fact, non-significant differences 
favoring the non-signaled condition were found in recall of topics in longer discussions. Similarly, they found that 
the non-signaled group was able to provide a greater amount of information about the topics they recalled than the 
signaled group, F(l, 71) = 5.06, p <.05 (p. 286). Torch et al. surmised that while signals improve recall of topics, 
particularly with short passages, they may do so at the expense of stimulating recall of topic-related subordinate 
information. In other words, signals may help the reader remember the organization of the passage but not 
necessarily the substance of it. 

Signaling effects on recall of main and subordinate topics. In two experiments involving 139 college students. 

Torch and Torch (1996b) asked students to read a 1,750-word passage that did or did not contain signaling devices 
(headings, overviews, and summaries) and complete a free recall task assessing recall of main and subordinate 
topics. The researchers hypothesized that signaling is most likely to have an effect on overall recall if the text is 
complex. In the first experiment, the simple topic structure (6 topics) was tested but non-significant differences 
were found. In the second experiment, which involved the more complex topic structure (12 topics), significant 
differences were found for recall of main topics, f(66) = 2.71, p < .05, and recall of subordinate topics, f(66) = 2.42, 
p < .05 (p. 42), favoring the signaled condition. Total recall was significantly improved as well, f(66) = 2.03, p < 

.05. Based on these results, signaling enhances overall recall when the reading passages are complex (12 topics) and 
do not necessarily enhance overall recall when the reading passages are simple enough for the student to encode the 
topic structure effectively without additional help. This is consistent with Spyrikakis and Standal’s (1987) study, in 
which signaling enhanced recognition memory only when the text was long and complex. 

On the surface, the findings from Torch and Torch’s (1996b) study run counter to their earlier study (Torch 
et al., 1993) in which signals were more effective with short, simple text. However, in the earlier study. Torch et al. 
(1993) operationalized complexity by manipulating the number of topic levels (2 versus 3), while in this later study, 
they altered the number of topics themselves (6 versus 12). Thus, comparing the two findings is problematic. One 
could draw the conclusion from these two studies that signaling is most effective with text that has no more than two 
levels, approximately 12 topic and idea units, and approximately 2,400 words. Text with less than 12 topics may be 
too simple to benefit from signals and, conversely, text with much more than 12 topics may be too complex to 
benefit from signals. 

Signaling effects on conditional recall. In a third experiment involving 120 college students. Torch and Torch 
(1996b) found that signals did not improve conditional recall (recall of subordinate topics when main topic is 
recalled), which rendered their impact on total recall non-significant. From this and their prior studies (Torch & 
Torch, 1995; Torch et al., 1993), Torch and Torch (1996b) conclude that while signals “influence the distribution of 
ideas and the organization of recall,” these gains “do not necessarily translate into better overall recall because 
signals do not consistently facilitate recall of subordinate content concerning a topic” (p. 45). Torch and Torch also 
found that half signals (i.e., including headings, overviews, and summaries for half of the topics but not for the other 
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half) improved recall of main topics for the signaled topics but depressed recall of non-signaled topics. In other 
words, signals appear to improve recall of signaled topics at the expense of non-signaled ones. This is consistent 
with Lorch et al.’s (1993) earlier findings. It is also consistent with a study by Robinson and Hall (1941), conducted 
primarily to compare comprehension scores and reading rates for different types of text (art, geology, fiction, 
Canadian and Russian history). They found that headings had non-significant effects on comprehension test scores. 

Do signaling effects improve with training? 

Headings ejfects on recall, structural knowledge, and recognition. Brooks, Dansereau, Spurlin, and Holley (1983) 
examined the effects of three signaling conditions (embedded headings, outline, and embedded headings with 
outline) on three dependent measures: recall (essay test), structural knowledge (outlining test), and recognition (28- 
item multiple-choice test). In experiment one, involving 132 college students. Brooks et al. found that headings had 
a significant effect on recall, F{\,\ 12) = 5.15, p < .02, and on structural knowledge, F(l, 112) = 6.41, p < .01, but 
not on recognition (p. 296). Headings improve recall of main ideas and knowledge of the organizational structure of 
the text. These results also suggest that recognition memory retrieval tasks, such as those often involved in 
multiple-choice and matching tests, are less sensitive to signaling effects. 

Headings ejfects on recall with and without training. In experiment two, involving 106 college students. Brooks et 
al. (1983) examined the effects of training students on the use of headings versus the effects of headings without 
training. Training significantly improved recall, F{2, 102) = 4.53, p < .01, and structural knowledge, F(2, 102) = 
19.90, p < .01, but again not recognition (p. 299). Either the multiple-choice test was not able to detect any existing 
signaling effects or recognition tasks are immune to signaling effects, even with training. 

Headings ejfects on free recall with and without training. Sanchez, Lorch, and Lorch (2000) also examined the 
impact of training on headings effects. In their study, 140 undergraduates read a 1,425-word text, then students in 
the training condition were instructed to form a mental outline of the text. This encouraged them to focus on the 
organization of the reading passage. Sanchez et al. (2000) found that students who neither had training nor headings 
had lower topic access scores (recall of main topics) and lower total recall scores than students in each of the other 
three conditions involving training and/or headings, f(136) = 3.05, p < .05 (p. 8). However, the effects of training 
and headings were essentially equal and did not interact to produce greater signaling effects. Access to subordinate 
information (i.e., conditional recall) was not affected by either training or the presence of headings. These results 
are consistent with prior studies (Lorch & Lorch, 1995, 1996a, 1996b) that found signaling effects for recall of 
organizational structure but did not find signaling effects for subordinate information. 

Signaling effects on problem-solving 

Signaling effects on free recall and problem-solving. In an experiment involving 58 tenth-grade college preparatory 
track students who were asked to read a 223-word passage of expository prose, Loman and Mayer (1983) found that 
headings and other signals caused students to modify their reading strategies, helping them recall more information 
and generate higher quality problem solutions. Students were given a reading passage that contained 43 idea units 
and, for those in the signaling condition, also contained preview sentences, underlined passage headings, and logical 
connective phrases. Idea units were blocked by type: conceptual, general, and primacy/recency (first idea/last idea). 
Loman and Mayer (1983) found significant differences between the signaled group (with headings) and the non- 
signaled group, /'’(I, 56) = 9.80, p < .01, and for the interaction of the three classes of idea units, F{2, 1 12) = 315.01, 
p < .001 (p. 406), suggesting that signals impact retention of different ideas disproportionately. 

Signaling ejfects for remedial learners on free recall and problem-solving. In a second experiment involving 44 
tenth graders who were designated as “remedial” and who read a 153-word passage containing 26 idea units, Loman 
and Mayer (1983) confirmed their earlier findings that the signaled groups remembered concepts and general idea 
units better than the non-signaled group. Conversely, they found that the non-signaled group retained more of the 
primacy/recency idea units. They concluded: “signaling was effective in asserting selective effects on what is 
learned” (p. 408). In other words, the paragraph headings, previews, and connective phrases drew the learner’s 
attention to the conceptual information, lessening their dependency on primacy and recency. 

In a study involving 32 college students and a 670- word passage dealing with the nitrogen cycle, Mayer, 
Dyck, and Cook (1984) examined the signaling effects of introductory definitions and headings on two criterion 
measures: a free recall test and a problem-solving test (10 essay or computational problems). The signals were 
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written to communicate the relationship among the main idea units, rather than merely the topic structure. 

Responses from the free recall test were grouped into four categories: (1) recall of conceptual information, (2) recall 
of primacy information (first 10 idea units), (3) total recall, and (4) verbatim recall of information. Responses from 
the problem-solving problem were grouped into two categories: (1) near transfer problems, which required specific 
information to solve, and (2) far transfer problems, which required the learner to use information in a creative way. 

Signaling effects of introductory definitions on recall of concepts versus verbatim information and far versus near 
problem-solving. In the first experiment, Mayer and his colleagues (1984) predicted that since the introductory 
definitions presented in the signaling condition were designed to communicate a relationship model, students would 
recall more conceptual information and perform better in problem-solving tasks at the expense of remembering 
verbatim and primacy information from the passage. The learners in the signaled condition did recall more 
conceptual information, f(30) = 3.84, p < .01, even thought total recall was not higher (p.l094). The signaled 
learners also recalled less primacy information, f(30) = 2.20, p < .05, and less verbatim information, f(30) = 2.54, p < 
.05 (p.l094). At the same time, they performed significantly better on the far transfer problem solving test than the 
non-signaled group, f(30) = 3.35, p < .05, confirming that introductory definitions facilitate more creative problem 
solving (p.l095. Signaling, however, had no effect on the near transfer problem solving items. 

Signaling effects of headings and introductory paragraphs on recall of concepts versus verbatim information and 
problem-solving. In experiment two, Mayer et al. (1984) examined the signaling effects of headings and 
introductory paragraphs. The study involved 30 college students and the same reading materials used in experiment 
one. Consistent with the first experiment, recall of signaled concepts increased, while recall of primacy and 
verbatim information decreased. Signaling also improved problem solving, f(28) = 2.10, p < .05 (p. 1098). 

Signaling draws learners to main ideas and reduces their dependency on verbatim memorization. By conveying a 
meaningful organizational structure of the content, signaling enhances problem-solving. 

Do verbal signals interact with non-verbal signals? 

Mautone and Mayer (2001) ran three experiments to examine signaling effects on two criterion measures, a 
free recall test and a transfer test. They began by examining signaling effects on recall and transfer. Then, the text 
was converted to audio narration so that the effects of audio signaling could be examined. Lastly, the researchers 
compared the effects of audio signaling with visual signaling, i.e., graphic animation. 

Signaling effects on recall and transfer. The first experiment involved 48 college students and a 487-word passage 
containing 7 main idea units dealing with the principles of airplane lift. The signaled version of the text passage was 
identical to the non-signaled version, except that the signaled version contained headings, a preview summary, and 
connecting words. The headings were bolded, underlined, and shown in a slightly larger (14-point) font, while the 
connecting words were boldfaced and italicized. The results indicated that while signaling did not significantly 
enhance recall, t(46) = 0.478, p = .63, it did significantly improve performance on a transfer test, t(46) = 2.15, p < 

.05 (p. 384). 

Signaling effects with audio narration. In experiment two, which also involved 48 college students, Mautone and 
Mayer (2001) used the same 487-word passage from experiment one, but this time it was presented in the form of 
audio narration without text. In the signaled condition, the intonation of audio headings and connective words were 
varied and an audio preview summary was added. The results indicated that this time, signaling significantly 
enhanced both recall, t(46) = 2.005, p < .05, and transfer, t(46) = 2.77, p < .01 (p. 384). 

Signaling effects with audio narration and animation. In experiment three, involving 86 college students, the same 
signaled and non-signaled audio narration were used, but this time, they were complemented with signaled and non- 
signaled animation. The animation was signaled using colored arrows as guides for selecting relevant parts of the 
illustration (as headings would); by using color to show the organization and relationships among the different parts 
of the illustration (as connective phrases would); and by including an icon that the student could click for a 
summary. The results suggest that signaled narration in combination with non-signaled animation enhanced recall, 
F{1, 82) = 4.01, p < .05 (p. 386), but that signaled narration was actually impaired by signaled animation. The 
transfer test results show a similar pattern favoring narration. When both signaled narration and signaled animation 
were used, students performed significantly better on the transfer test, f(44) = 2.76, p < .01. However, signaled 
narration significantly improved transfer, /^(l, 82), p < .01 (p. 386), while signaled animation appeared to have 
almost no impact. 
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Mautone and Mayer (2001) concluded that verbal signaling (text and audio narration) significantly 
enhanced transfer to new problem-solving situations, but had only slight to moderate effects on recall. Animation, a 
non-verbal signal, had little or no positive impact on recall or transfer. Their findings suggest that verbal signals are 
stronger than non-verbal ones, and that when used simultaneously, they can compete for attention and for short-term 
memory space. 



Summary 

When headings and other signals are absent from instructional text, learners must rely on other sources of 
information to identify the key points covered in the reading passage. These sources include the learner’s prior 
knowledge as well as the serial positioning of topics (e.g., important topics first and less important topics later) and 
the amount of elaboration provided in the passage (e.g., important topics are explained more thoroughly). Signaling 
devices explicitly highlight the key points of a passage and communicate the structure of the text. 

When are signals most effective? Signals do not necessarily increase the total amount of information 
recalled. Rather, they mainly alter the learner’s reading/encoding strategy (Loman & Mayer, 1983). Several studies 
have shown that signaling significantly improves recall of main topic words and phrases (Brooks et ak, 1983; 

Hartley et ak, 1980; Krug et ak, 1989; Loman & Mayer, 1983), with both cued and free recall tasks. Signals, 
however, do not necessarily improve recall of subordinate information (Torch & Torch, 1996a). In fact, headings 
and other signals shift the focus of attention during encoding to the signaled information and away from the non- 
signaled information (Krug et ak, 1989; Torch & Torch, 1996b; Robinson & Hall, 1941). Although signaling may 
not improve recall of subordinate information, it does enhance transfer of knowledge to new situations, thus aiding 
in problem-solving tasks (Mautone & Mayer, 2001; Mayer et ak, 1984). Signaling is an appropriate strategy when 
recall of the cued topics is important and when the key topics are communicated to facilitate problem-solving. 

Multiple-choice, matching, and other recognition item formats appear to be less sensitive to signaling 
effects compared to cued recall and free recall tasks (Christensen & Stordahl, 1955; Marcinkiewicz & Clariana, 

1997; Spyridakis & Standal, 1987). Smith’s (1986) outshining hypothesis posits that environmental cues enhance 
memory retrieval when the to-be-remembered information is difficult and do not when it is easy to remember. 
Recognition tasks are generally easier than recall tasks, and so this may explain why signaling is less effective with 
multiple-choice tests than it is with recall tests. The difference may also be due to ceiling effects (Spyridakis & 
Standal, 1987), which are more likely to influence easier test formats. 

Statement headings and question headings are equally effective (Hartley et ak, 1980). Headings are not any 
more effective than other signaling devices (Torch & Torch, 1995). However, headings are highly effective when 
grouped with other verbal signals (Brooks et ak, 1983; Hartley et ak, 1980; Loman & Mayer, 1983; Torch & Torch, 
1996a, 1996b; Torch et ak, 1993) but are not more effective when combined with non-verbal signals (Mautone & 
Mayer, 2001). 

Signaling effects are strengthened when the information is cued at retrieval (Torch & Torch, 1995). Signals are e 

specially helpful when the text is approximately 2,400 words in length and has approximately 12 topics 
with no more than two levels (Torch & Torch, 1996b; Torch et ak, 1993). Text that is too short and simple or too 
long and difficult may not benefit from signaling effects. 

Implications and Conclusion 

Instructional designers and educators commonly use headings and other signals when writing instructional 
text for paper-based and screen-based materials, regardless of the instructional objectives. The research suggests 
that headings and other signals can be powerful textual cues that alter the encoding strategy of learners. While 
headings can promote recall of main topics and enhance problem-solving, they are not helpful in promoting reading 
comprehension. In fact, signals tend to cause learners to remember key ideas at the expense of subordinate 
information that may be crucial to understanding. This implies that writers of instructional text should consider the 
kinds of instructional objectives sought before deciding whether or not signaling will be helpful or counter- 
productive. 

If headings are being used for the purpose of enhancing recall, they will be most effective when used in 
conjunction with other verbal signaling devices, such as introductions, summaries, transition statements, and titles. 
Headings will be most useful when the text is moderately long and complex, and when comprehension of the 
subordinate information is less important than understanding of the main points. 

Much of the signals research involves paper-based instructional materials but few involve screen-based 
materials. In online venues, instructional text can assume complex forms, with text hyperlinked to subordinate or 
related content, so the text is not always sequentially ordered, and elaboration is dependent on the user’s desire to 
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explore by clicking to reveal deeper layers of information. In such settings, would verbal signaling devices be more 
helpful or less helpful in promoting recall, comprehension, and problem-solving? Given the increasing use of non- 
verbal cues, such as icons, graphics, animations, and color, do they compete with or else complement verbal signals 
in different virtual settings? Given the proliferation of computer-based and web-based courseware, online distance 
learning, and electronic textbooks, future signals research should reevaluate the effects of headings and other signals 
in the digital medium to provide designers clearer answers to such questions. 

Headings are valuable signaling devices that have been shown to promote recall of main ideas, knowledge 
of organizational structure, and problem-solving. They draw the learner’s attention to the key topics in a reading 
passage. These key topics form the topography of the text based on the organizational structure of the passage. As 
such, they implicitly contain relationships and other structural information. Tasks and skills that benefit from this 
improved structure of the content are therefore likely to benefit from signals. However, it is critical to recognize that 
these gains are made at the expense of memory for subordinate information. 
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Abstract: 

The purpose of this study is to examine the effectiveness of Internet filtering software, the current use of 
filtering software in Colorado public libraries and the costs public libraries would incur should Internet filtering 
software become a state mandate. First, the Internet is briefly discussed, including the beginnings of this global 
information superhighway, as well as its original intent. Second, the important issues surrounding Internet access in 
public libraries are presented. Third, a discussion of Internet filtering software including some of the products 
available and how the software works is presented. Finally, the results of a survey of Colorado public libraries’ use 
of Internet filtering software are discussed, including recommendations for the future. 

Introduction to the Internet: a brief history 

The Internet is a series of computers connected together to form a “web” or global network of computers. As a 
result of this connectivity, files and pages can be downloaded to any computer with a connection to the network 
(Barker, 2002; http://www.pcwebopedia.com . 2003). Millions upon millions of pages are available on the Internet, 
and anyone with access to server space can create a page, post it, and link it to others (Barker, 2002; 
http://www.pcwebopedia.com . 2003). 

The Internet began as a United States Department of Defense network in the 1960s that would continue to 
function in the event of a disaster, such as a nuclear war. The original network, connecting scientific and academic 
researchers, was called Advanced Research Projects Administration Network (ARPANET) (Kirstein, 1998). In 
1985, the National Science Foundation created NSFNET, which was a series of networks designed to allow 
educators and researchers to communicate. Based on the protocols developed by ARPANET, a backbone network 
was created to link any U.S. educational or research institution (Gaston, 1995). 

Software to make access to the network clearer and more user- friendly was created and telecommunication 
companies began to build their own networks to make access to the global network more convenient (Hardy, 1993). 
As faster computers have become more affordable over time and are now available to a larger percentage of the 
population, Internet use has grown dramatically (Bushong, 2002; Hardy, 1993; Leiner et al, 2000). According to the 
2000 census, 54 million homes (51%) reported having one or more computers, and 44 million households (42%) 
reported at least one person using the Internet at home. Of children age 3 to 17, 65% lived in a house with at least 
one computer, and 30% of the children use the Internet at home (U.S. Bureau of Census, 2000). In a study of 
computer users in the United States, 10% of Internet users reported their online activities take place at a public 
library (U.S. Department of Commerce, 2002). 

What is Internet filtering software? 

The primary way of preventing end-users from accessing objectionable material is increased adult involvement 
in children’s online activities (Bushong, 2002). However, this is not always feasible in a public library. Following 
recommendations for Internet safety that suggest putting Internet terminals in high-traffic areas (Bushong, 2002), 
libraries have tried to arrange computer banks so monitor screens face into the center of the room and are not 
isolated. This solution still requires constant supervision by library personnel, and can be costly in the amount of 
time required to properly monitor this area so public libraries must rely on technological solutions (Bruce, 1999). 

There are two ways Internet content can be filtered for the end-user: either a hardware solution, such as a 
filtering “box” that scans material before it ever reaches the screen of the end user; or Internet filtering software, 
which works with an Internet browser to block certain types of pages and/or information that is labeled as 
undesirable, inappropriate or illegal as it reaches the end-user’s computer. There are dozens of software packages 
commercially available, ranging in price from $20 to well over $2,000, depending on features, the number of users, 
and whether it is terminal- or server-based (Bruce, 1999; Smith, 2000). 

Solutions fall into several categories: (a) server-side filtering software using a specific URL list for blocking, 

(b) terminal-based filtering software using a browser that will not allow certain types of material to load, (c) 
terminal-based filters that block certain text, and (d) filtering software that uses labeling and rating systems and will 
not allow material that is not within acceptable limits to load (COP A, 2000). 
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In a 2001 nation-wide study, 44% of public libraries indicated they use some form of filtering software, but 
those libraries that filter only some computers indicate the filtered computers are only in the youth areas (Curry & 
Haycock, 2001). Approximately 69% of public libraries that do not filter had no plans to install any filtering 
software in their library. Most school libraries that use filtering software, while for the most part happy with the 
software’s ability to block the objectionable sites, are dissatisfied at some level with the software’s ability to allow 
appropriate sites through the filter (Curry & Haycock, 2001). 

How Internet filtering software works 

As stated previously, filtering software can block content based on forbidden words, a list of forbidden websites 
(URLs) or protocols (such as e-mail, chat rooms or bulletin boards) (Consumer Reports, March 2001; Smith, 2000). 
The software works in conjunction with a web browser, such as Microsoft’s Internet Explorer or Netscape 
Navigator, to block Internet content by previewing content before the page is loaded by the local computer (Barker, 
2002). Some Internet blocking software packages block as few as six categories, but some block many more (Curry 
& Haycock, 2001; Edelman, 2001; Hunter, 2000). Some of the blocked categories may include intolerance, 
violence/profanity, nudity, sexual acts, satanic/cult, drugs/drug culture, hate speech, criminal activity, sexually 
explicit speech, “adult” speech, violent speech, religious speech, and sites that have a link to secure sites such as e- 
commerce sites, sports or entertainment (Hunter 2000; Lipow 2000). 

Several different studies of different Internet filter software packages have determined that the software 
packages studied were ineffective, by either under-blocking content (allowing objectionable material through the 
filter) or over-blocking content (not allowing appropriate and useful sites to load) (Hunter, 2000; Janes, 2002; 
Lipow, 2000; Nabel, 2000; Schneider, 1997). 

Typical inappropriately blocked sites include terms such as “xxx” (which also blocks all sites that talk about 
Superbowl XXX) and sites containing any mention of specific parts of the human body (which also blocks sites 
about chicken breast recipes or sites about breast or testicular cancers). Sites that discuss feminism or gay and 
lesbian issues are often blocked, while sites that oppose homosexuality are not blocked (Hunter, 2000). Pages with 
pictures might be inappropriately blocked as well - a photo titled “adult.jpg” depicting an adult sitting on a park 
bench would be blocked because of the “adult” designation (Consumer Reports, March 2001) while another photo 
with an unblocked name containing nudity would not. 

Additionally, some Internet filtering software will not allow access to sites that the Internet filtering software 
company has not rated (Edelman, 2001). With new sites and pages being added to the Internet daily, filtering 
software is unlikely to be completely up-to-date. These filters often block appropriate sites because the filtering 
software often does not consider the context in which a word or phrase is used. 

It is far more troubling when a filter appears to block legitimate sites based on moral or political value 
judgments (Consumer Reports, 2001; Rideout et al, 2002). As an illustration of an Internet filtering software’s 
performance, it should be noted that the travel site for Castle Rock, CO, is blocked by at least one of the filtering 
software packages studied (Edelman, 2001). Many of the relevant studies about Internet Eiltering software can be 
found summarized in Appendix C. 

What issues are important in public library Internet access? 

What is a public library? Is a public library a safe haven where parents can send their children, knowing they 
will be protected from controversial issues; or a place of free speech where many types of materials are available to 
the public? 

When the Internet came into widespread use, it became necessary for public libraries and schools to provide a 
connection to this “information superhighway.” Recent federal initiatives emphasize the importance of using 
technology in schools. The Schools and Libraries Universal Service Support Mechanism (known as E-rate) allowed 
schools and libraries across the nation to receive discounted telecommunications and Internet services (Revenaugh, 
2002; USAC, 2002). Nationwide, approximately 95.7% of public libraries offer Internet access and nearly all have 
some sort of acceptable use policy (Estabrook & Lakner, 2000). 

Filtering Laws. With the growing number of “inappropriate” pages available on the Internet, Congress has 
passed a variety of laws attempting to limit the content made available in public places on the Internet (Samoriski et 
al, 1997). The Communications Decency Act of 1996 was passed by Congress but was struck down by the Supreme 
Court on June 26, 1997 on Eirst and Eifth Amendment (Madison, 1791) grounds in a 7-2 decision (ACLU v. Reno, 
1996). 

Congress next passed the Child Internet Protection Act (CIPA) of 2000, which ties library funding to the 
mandated use of filtering software on Internet terminals that could be used by both adults and minors in public 
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libraries (H.R. 4577; Public Law 106-554). Because libraries are funded by state and federal funds, fund reception 
was tied to compliance with CIPA. Libraries that did not comply with CIPA by July 1, 2002 would need to decline 
their government sponsored funding and discounted Internet access under E-rate. Libraries accepting the funding 
needed to submit certification by October 28, 2002 that steps had been taken to comply with CIPA. However, the 
United States District Court for the Eastern District of Pennsylvania found that CIPA was “facially invalid” under 
the Eirst Amendment to the United States Constitution (American Library Association et al v. United States of 
America et al, 2002; Multnomah County Public Library v. United States of America et al, 2002). The Supreme 
Court of the United States has agreed to hear the government’s appeal and a hearing has been set for March 5, 2003. 

Public library attempts to conform with current legislation to restrict Internet access to its patrons have also 
been called into question (Mainstream Loudoun v. Board of Trustees of Loudoun County Library, 1998). Eiltering 
software was used in public libraries before federal mandates were installed. However, many of those restrictions 
have been successfully challenged. In Loudoun, a federal court in Virginia found that a library’s use of an Internet 
software filter violated the Eirst Amendment. Several experts offered testimony in this case (Janes, 1998; Lipow, 
2000; Edelman, 2001), indicating that too many sites of legitimate use and interest were being blocked by one or 
more Internet filtering software packages (Doyle, 2002; Eysenbach & Diepgen, 2001). While software has improved 
since Loudoun (James, 1998), there are still sites blocked by Internet filtering software that are of legitimate interest 
to the general public (Newberger, 2001; Smith, 2000; Edelman, 2002). Eilters are unable to block 100% of Internet 
content that falls into certain categories while simultaneously allowing access to other similar but appropriate 
content (Expert Report, Edelman, 2001; Rebuttal Report, Edelman, 2001). A 2002 study found that when simulating 
health searches using several different available Internet filters at different restriction levels, access to between 1 .4% 
and 25% of appropriate health sites were denied, depending on the level of restriction at which the filter was set 
(Rideout et al, 2002; Minkel, 2003). 

Use of Internet filtering software in Colorado Public Libraries 

The purpose of this study is to examine the effectiveness of Internet filtering software, the current use of 
filtering software in Colorado public libraries and the costs public libraries would incur should Internet filtering 
software become a state mandate. The research questions examined were: 

1) What is the cost to a library jurisdiction to filter? This includes not only the cost for the software, but 
also the cost in hardware, staff time and staff training. 

2) Is there a relationship between the cost and effectiveness of Internet filtering systems? 

3) How many Colorado public libraries currently use Internet filtering software? 

Methodology 

A survey was sent to all public library jurisdictions in Colorado according to a list generated by the 
Colorado State Library. The survey asked for the name of the library jurisdiction to determine participation. 
Questions included whether the library jurisdiction owns at least one computer with capabilities to run most Internet 
filtering software (to rule out deficient equipment questions) and whether the library jurisdiction is filtering all or 
part of their Internet access terminals. Other questions asked for the name(s) of filtering software being used, the 
cost of the software, related equipment, and staff. Einally, several options were given as reasons for the decision to 
filter or not filter in that particular library jurisdiction. The survey was sent twice to all respondents either as an e- 
mail attachment or by the United States Postal Service. Additionally, a message was posted to the Public Library 
Director’s listserv, asking for participation. 

Results 

The population for this study is the public libraries in the state of Colorado. Of 1 16 public library 
jurisdictions in the state, 54 surveys were returned either via e-mail attachment or a pre-stamped envelope, yielding 
a 46.6% response rate. The responding library jurisdictions were 53.7% small libraries (15 computers or less) and 
46.3% large libraries (more than 15 computers) Because library jurisdictions have varying numbers of library 
buildings, the survey was given to library directors and included from one building to the largest library jurisdiction 
in Colorado, containing 23 branches. Data was obtained through the use of a voluntary survey that was sent to and 
returned by the respondents; the sample was obtained by library jurisdictions that chose to participate. It should be 
noted that the survey was sent out less than one month after the required annual survey administered by the 
Colorado State Library, which may have adversely affected the response rate for this study. 

All respondent library jurisdictions reported having at least one IBM-compatible computer with a minimum 
level of RAM, hard drive space, operating system and processor speed necessary to run most Internet filtering 
software packages. Eleven of the respondent library jurisdictions reported having at least one Macintosh computer 
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as well as at least one IBM-compatible computer (see Figure 1) and there were two non-respondents. There were no 
Macintosh-only environments. The number of computers in each library jurisdiction ranged from two in the smallest 
library jurisdiction to 760 in the largest responding library jurisdiction. 



Type of Computer 




10 20 30 40 

Frequency of Response 



50 



Figure 1: Type of computer as reported by responding library jurisdiction 



Of the respondent library jurisdictions, 12 (22.2%) reported having some level of filtering software in their 
libraries (see Figure 2), and 41 (77.8%) report not using filtering software. Of those that filter, two-thirds are large 
(more than 15 computers) libraries. 



Category 


Frequency of response 


Percent 


Filter all computers 


7 


58.3% 


Filter some computers 


5 


41.7% 


Filter all computers in the 
children’s area 


8 


66.7% 


Filter some computers in the 
children’s area 


1 


8.3% 


No computers in the children’s 
area 


4 


33.3% 



Table 1: Summary of Key Findings for Filtering Library Jurisdictions 



Five of the filtering library jurisdictions were required to purchase equipment not already available at a cost 
of over $1000 per library jurisdiction. Filtering software was installed in those libraries as early as 1997 and as 
recently as 2001 (see Table 1). Of those twelve, nine (75%) of the filtering library jurisdictions filter some or all of 
the computers in the children’s area in the library. The libraries that report not filtering any of their computers 
include the smallest responding library jurisdiction (two computers) and the largest responding library jurisdiction 
(760 computers) with a median of 14 computers. Four of the total respondent library jurisdictions (7.4%) report that 
the computers in the children’s areas are not Internet-accessible. 
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Year Installed 


Number of installations 


1997 


1 


1998 


1 


1999 


3 


2000 


2 


2001 


1 



Table 2; Installation year of Internet filtering software 

Filtering Libraries. Libraries reported using a variety of Internet filtering software packages or 
arrangements. Twenty-five percent of filtering library jurisdictions use a server-side filter with varied levels of 
filtering (usually three to five levels). Access to different levels is obtained by an automated card reader or a 
regularly changed password obtained at any of the library’s desks. For libraries that use a permission or password 
system to circumvent the filtering system, parents can either sign Internet use forms or obtain the password for their 
minor child to allow them full access. Library jurisdictions that require a password do not question an adult 18 years 
of age or over who requests the password. Three of the libraries that filter only part of their computers report the use 
of privacy screens to prevent unauthorized persons from seeing potentially inappropriate material. 

Several reasons were offered for the decision to filter Internet connections, which included CIPA, 

Director’s decision. Board Policy, Community Request, and Incidence of Misuse, which are summarized in Figure 
2. In addition to the given reasons, several library jurisdictions used the ‘other’ option to give the following reasons 
that did not fit in one of the given categories. Their responses were as follows: 

• Filtering was originally recommended by the State Grant. 

• Request for filters was initiated by a single member of the City Council. [We] were not required to filter 
until Internet computers became available in the children’s room. We have purchased privacy screens for 
the two unfiltered computers at about $230 each. 

• School wanted to do it. 

• Two computers were purchased and maintained by the *** school district, and they have the filter. 

• Basically the grant was written so we would be connected to the school district’s computers so that’s how it 
turned out. 
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Figure 2: Reasons for Internet filtering, as reported by libraries 



308 



Non-filtering library jurisdictions. Several reasons were offered as reasons for the decision not to filter Internet 
connections, including Director’s Decision, No need/No problems. Board Policy, and Community Request, which 
are summarized in Figure 3. In addition to the provided options, several libraries that report not filtering used the 
‘other’ option to give the following reasons that did not fit in any of the given categories. Their responses were as 
follows: 

• Cost was prohibitive, especially in terms of staff time. VFe have minimized the problem through use of 
privacy screens, PC placement, policies, web design and family training. 

• The decision not to filter was made by the Director with the knowledge and support of the Board. We 
perceived “no need, ” not from a complete lack of problems, but from very few problems. Of course there 
have been some problems. Our patrons are human, after all. 

• Computers were done on the Bill Gates grant to libraries from the previous director. She never felt the 
need for the filtering system. In addition, [there was] no budget. 

• We visually monitor all public access machines, which are located in full view of [the] circulation [desk], 
[the] reference [desk], [and the] children’s librarian’ s desk. FILTER’S DON’T WORK! And, by all means 
not incidentally, we believe in the First Amendment and regard its defense as our number one priority. 

• Freedom of information; also, I do not believe filtering actually works. Materials are filtered which 
shouldn’t be (fine arts, health) and not filtered that should be. 

• Our IT Department tested them and they don ’t work! Therefore, we didn ’t want to provide parents with a 
"false sense of security. ” We decided not to allow Internet on the computers in our Children ’s Room, 
because we often have just one person working in the library and cannot see what is going on in our 
downstairs Children ’s Room. We have an Internet Use policy in place of filters. 

• Town of 500, easy to monitor - computers right next to director’s desk! 

• Gates computers are very proprietary and we decided - not worth it. We use parental permission instead. 

• [We] believe [filtering to be] unconstitutional. 

• Public Computer Use Policy that is in compliance with CO state statute, and CIPA, is posted by each 
computer, clearly stating that violation of rules of use results in loss of library privileges. Reference sites 
for students have been book-marked. The computers screens are within 15 feet of and facing librarian’s 
desk. In seven years there has only been one violation. 

• [The] librarian monitors the computers. 

• It does not work & it is a form of censorship 
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Figure 3: Reasons for not filtering, as reported by libraries 
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Internet use policies. According to the Colorado State Library, Internet use policies were required as part 
of state funding. Each library has an appropriate Internet use policy on file with the Colorado State Library. 

Cost of filtering. The cost of filtering varies depending on the features of the chosen filtering software or 
equipment, the extent of filtering desired by the library and whether the filter is software-based (on each individual 
computer terminal) or server-based (at a distant location that filters the Internet connection before it reaches the 
computer terminal). Table 2 represents the initial cost of filtering per library for the eight libraries that provided this 
information (Please note: library jurisdictions are identified by case number.) 



Jurisdiction 


Initial Cost 


27 


$49.99 


13 


$64.00 


32 


$182.48 


46 


$350.00 


25 


$450.00 


1 


$600.00 


9 


$6,000.00 


41 


$44,000.00 



Table 2: Initial Financial Impact of filtering 



Five of twelve responding filtering library jurisdictions chose a server-based filter system. One of those 
elected to institute a ‘SmartCard’ system, which included significantly more equipment than just the server 
necessary to institute filtering on all computers in the library. The other four that elected server-based filtering 
products faced an average financial outlay of $5281.00 for the server and the software package. Costs for the service 
alone varies depending on the number of computers being filtered; servers will generally run between $1000.00 and 
$2000.00, depending on the needs of the library and other functions the server is to perform. The other four libraries 
reported a cost of between $49.99 and $600.00 per computer for terminal-based software, with an average cost per 
computer of just under $303 per computer. 

In addition to the initial cost, there is also an annual cost for current subscriptions to the service and updates 
or yearly maintenance. Figure 5 depicts those costs for the seven library jurisdictions that provided this information. 



Jurisdiction 


Annual costs 


27 


$39.99 


25 


$450.00 


7 


$1,500.00 


32 


$1509.00 


9 


$2231.00 


24 


$3,000.00 


41 


4,300.00 



Table 3: Annual subscription and maintenance cost of filtering 

For the library jurisdictions electing to use a server-based filtering package, the average annual 
maintenance and subscription service is approximately $3177.00. Terminal-based software is not run on a server that 
requires yearly maintenance, but does require a subscription renewal for software updates. The average annual cost 
as reported by Colorado libraries for subscriptions is $874.75 per computer. 

Library jurisdictions that filter their Internet connection were also asked about the time spent training staff 
to monitor patron use of the Internet filtering software, such as time spent having to activate the software for a minor 
and deactivate it for an adult. The responses are displayed in Figure 4. While for most (67%) of filtering libraries, 
the cost associated with staff time was minimal, 17% of the filtering libraries reported more than $2000 spent in 
staff training time to monitor Internet filtering software, and 16.7% report the expectation that staff training relating 
to their Internet Filtering Software to continue as a yearly expense. 
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Figure 4: Cost in staff training for Internet filtering jurisdiction 



Conclusion 

Based on data from the study and comments made by individual library directors in the state of Colorado, it 
appears that to impose a mandatory Internet filtering policy in Colorado public libraries would impose financial, 
ethical, and legal hardship for public libraries across the state. 

Should a mandatory filtering law be passed in Colorado, several problems would develop. First, libraries 
dependent on government funding would face the huge financial hurdle of purchasing appropriate hardware and 
software. Many of the terminal-based software programs, while seemingly more reasonable in initial outlay, are not 
as effective, and many of the libraries using these programs report a sizable use of staff time to turn the filtering 
software on and off for patrons of different ages. Smaller libraries that do not have a server would be required to 
purchase one to institute the more reliable server-side software, which is more expensive than traditional terminal- 
based software. Not including the cost in staff time, the appropriate budget per computer for a reasonably priced 
server and server based-software, including yearly subscriptions, reaches nearly $2,000 per computer. Subscriptions 
for terminal-based software can range from less than $30 per terminal to more than $100, but research has shown 
most of these to be ineffective, by either under-blocking content or over-blocking content. 

Second, the ethical debate on the definition of an American public library would intensify. Public libraries 
should not be responsible for filtering information deemed “appropriate” by one individual. While libraries can and 
do use reasonable judgment in allowing minors to access inappropriate material, it is the constituents of that library 
district who should have the final say about what is kept in the library their tax dollars support; the decision to filter 
needs to be controlled at the local level. It would seem reasonable to require filtering on all Internet-accessible 
computers in the children’s area only or to require filing with the state an appropriate Internet Use Policy that all 
patrons or legal guardians of patrons would be required to sign, but a mandatory filtering law could very easily be 
challenged on its merits. 

Third, a legal battle could ensue if libraries are required to filter all Internet access. Should the decision by 
the District Court of Eastern Pennsylvania be upheld by the Supreme Court and CIPA overturned, libraries would 
have to discontinue use of filtering software or risk legal action for violating their constituents’ First Amendment 
rights. In addition, it is reasonable that the State of Colorado would be named as a defendant in any such legal matter 
for passing a law that is unconstitutional. The United States Constitution and the First Amendment cannot be 
ignored: ‘'Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise 
thereof; or abridging the freedom of speech, or of the press', of the right of the People to peaceably assemble, and to 
petition the Government for a redress of grievances (emphasis added) (Madison, 1791).” As stated above in the 
review of the literature, all other laws regarding the prevention of access to information in a public library have 
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come under scrutiny and been overturned by the Supreme Court, and there is no reason to believe CIPA will be an 
exception. It seems prudent to delay the passage of any state mandate regarding the use of Internet filtering software 
or regarding any related issues until the Supreme Court has acted on CIPA. 

Professor Lawrence Lessig, noted cyberspace law expert at Stanford University Law School, stated that 
mandatory filtering is not the answer. While parents may be able to prevent their children from accessing 
inappropriate Internet content, states cannot become censors for children (1998). Any attempt at mandatory Internet 
filtering in any state would be nothing less. 
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PFnet Translation: A Tool for Concept Map Quantification 



Debbie Denise Reese 
Wheeling Jesuit University 

Abstract 

Concept maps provide an attractive external representation of knowledge, but they are idiosyncratic and 
therefore difficult to compare across studies and educational assessments. Translation into Pathfinder Networks 
(PFnets) is one approach to solving this problem. By calculating the distance between all the pairs of nodes within a 
given concept map, researchers and evaluators can isomorphically translate a concept map into a Pathfinder 
Network (PFnet). Thus quantified, concept maps can be compared (a) with a normative domain model, (b) across 
learners, and (c) longitudinally to measure changes in understanding over time. Maps can be aggregated, allowing 
cross-sectional or longitudinal comparison of groups. The author describes the translation method and provides 
examples of translations and analyses. Data are protocol responses generated by fifty-seven fourth and fifth-year 
preservice elementary teachers after completing a unit of metaphor-enhanced, computer-mediated instruction. 

Introduction 

In practice, assessment through concept mapping has been inconsistent and riddled with considerable 
variety in (a) the individual subscores employed for scoring concept maps, (b) the underlying theoretical rationale 
for subscore selection, and (c) subscore weighting scales (Edmondson, 2000; Markham & Mintzes, 1994; Novak, 
1990; Novak & Gowin, 1984; Wallace & Mintzes, 1990). Evaluators score any combination of the number of 
conceptual branches, hierarchical levels, intersections between branches (cross-links), and links between concepts 
(propositions). Unfortunately, some subscore and weighting combinations reward learners for superficial knowledge 
structures. Additionally, although concept map scorings measure the richness of mental models, they do not readily 
afford comparisons of the structure of learners’ mental models. In contrast. Pathfinder networks (PEnets) appear to 
provide a rigorous and unbiased tool for model comparisons. PEnets derive from graph theory, a branch of 
mathematics (Schvaneveldt, 1990). PEnets contain nodes (concepts) and the connections between nodes (relational 
links). A network is a set of nodes and links. When constructed, PEnets look very much like concept maps: ovals 
connected by lines. There is one big difference between the two. PEnets are mathematical entities. As such, they are 
inherently composed of quantities and relationships. They are sets of numbers. Therefore, PEnets can be accurately 
compared over time, across students, or to an expert model. They can be statistically analyzed. Accurate and 
consistent translation of concept maps into PEnets might reduce the reliability or accountability problems associated 
with concept map scoring. This paper describes a method for translating concept maps into PEnets as well as an 
example of PEnet analysis of protocols produced within an empirical study of metaphor-enhanced, computer- 
mediated learning environments. 



The Structure of the Concept Map 

Psychologically, people view knowledge domains as systems of objects, object-attributes, and relations 
between objects (Gentner, 1983, p. 156). Visual tools, such as concept maps, represent knowledge domains 
graphically (see Eigure 1). Within a concept map, the object is a concept; it is represented as a labeled oval. Arcs 
represent relations between concept dyads. 

Joseph Novak and his colleagues (Novak, 1990; Novak & Gowin, 1984; J. D. Novak & Musonda, 1991) developed 
the concept map as a research tool for assessing changes in learners’ mental models resultant from instruction. The 
structure and characteristics of the map derived from David Ausubel’s (1962; 1963) assimilation theory. The theory 
posits that knowledge structures are hierarchical, with superordinate concepts subsuming subconcepts. In addition, 
people organize knowledge through two cognitive processes. During progressive differentiation, learners sort and 
gather domain subconcepts into conceptual strands. During integrative reconciliation, learners construct relational 
connections between disparate conceptual strands. Within a concept map, dyads of two concepts and the relationship 
that joins them form a proposition (Gentner, 1980, 1983; Novak & Gowin, 1984). The number of propositions 
within a concept map measures the extent of an individual’s domain knowledge. The other subscores measure 
degrees of processing evidenced within a concept map: number of hierarchical levels measures subsumption, 
number of conceptual branches measures differentiation, and number of cross-links measures integration. Well- 
developed mental models of targeted domains contain greater numbers of propositions connected through 
subsumptive hierarchy, progressive differentiation, and integrative reconciliation. Mental models held by domain 
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experts and highly successful domain learners evidence well-developed hierarchical structure, differentiation, and 
integration (Edmondson, 2000; Fisher, 2000), and “domain-specific knowledge has been shown to be a critical 
component of competent performance” (Fisher, 2000, p. 203) as well as an accurate domain-specific predictor of an 
individual’s ability to problem-solve (Gordon & Gill, 1989). 

Reese (2003a) has demonstrated that, in at least one domain (educational philosophy; functionalism), 
concept map cross-links are more sensitive indicators of well-developed mental models than the other subscores. 

The number of propositions follows as the second most sensitive subscore. There is very little difference between 
the number of hierarchical levels or differentiation branch subscores evident in concept maps when concept maps 
constructed from undeveloped mental models are compared to concept maps constructed from well-developed 
mental models. Relative subscore sensitivity suggests that weighting scales should reward cross-links and 
propositions. While summed subscale scorings afford evaluation and comparison of the richness of learners’ mental 
models, the idiosyncrasy of individual maps precludes structural comparison (a) to an expert model, (b) across 
learners and (c) over time. Inter-map comparison of relational structure requires a quantified, relationally isomorphic 
equivalent and a translation methodology. 

A Quantified Concept Map 

The concept map illustrated in Figure 1 is the graph of a Pathfinder network produced using PCKNOT 
software, the knowledge Aetwork Organizing Tool for the IBM PC (McDonald, Schvaneveldt, & Sitze, 1998). 
PCKNOT uses an algorithm to calculate a Pathfinder network (Schvaneveldt, 1990) from relational specifications 
and quantification of their strengths. The graph consists of set of nodes and all the links that join node dyads. Nodes 
are analogous to concept map concepts and links are analogous to concept map relations. 

Within a network, a path is a journey through a set of linked nodes, from an initial node to a terminal node. 
The path moves through adjoining concepts and relations (like a baseball player moving along the bases, with the 
bases as analogs for nodes and the baseline segments as links). For example, the path (pathi) from ascribed 
characteristics to informal education in Figure 1 actually moves according to the path illustrated within Figure 2: 
ascribed characteristics (node) leading to (relational link) social change (node) evidenced in (relational link) 
informal education (node). The length of a path corresponds to the sum of the number of unique links traversed in 
moving from the initial node to the terminal node. The length of path' is 2. 

A graph represents the structure connecting a set of nodes and can be displayed as an image, like the one in 
Figure 1 . When positive, real numbers are assigned to each link in the set, the graph becomes a network. The weight 
of a path is the sum of the weighted links. PCKNOT calculates the geodesic for each of the set’s node dyads; that is, 
it calculates the shortest possible path between each of the nodes. Thus, for a set of nodes and links, a network 
specifies structure (the graph) and quantifies it (the weighted geodesics). If the weight of every link in a network 
equals one, then the network equals the graph. Although Figure 2 is relationally equivalent to Figure 1, it looks 
different for two reasons; (a) the translation methodology is based on setting the link weights to one and (b) 
PCKNOT drew the relational links to scale. When plotting consists of equidistant links, the characteristics of this 
expert model cause the nodes to superimpose, as in Figure 1 . Standardization of link length within the PFnet 
constrains the PCKNOT placement of concept nodes such that informal education and role differentiation must 
overlap on the graph. PCKNOT software will allow the researcher to drag the graph nodes that overlap while 
maintaining the relational links. A researcher could drag the nodes in Figure 1 to make it look exactly like Figure 2. 

Relational Linkages 

Originally, concept map propositions consisted of labeled concepts and unlabeled relational links. Over 
time, Novak and his colleagues developed mapping conventions; (a) relational links are labeled and (b) relationships 
are assumed to be directional, flowing from top to bottom (Novak & Gowin, 1984). Pathfinder networks do not 
specify the semantic content of the links (Schvaneveldt, 1990) and the software allows for the specification of both 
bi-directional (two-way) and unidirectional (one-way) paths. Other semantic modeling software system may require 
slightly different input. For example, SemNet/Semantica software (Fisher, 2000; Semantic Research Inc., 2002) 
requires specified bi-directional links, relational labels, and definitions of both concepts and relations. 

Translation Methodology 

Translation from concept maps into PFnets is based upon two PFnet characteristics; 

1 . The graph equals the network when the weight of each link within the network equals one; thus, a graph is a 
network in which all links are weighted at one. 

2. The length of a path from initial to terminal nodes is the sum of the number of intervening links. 
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Therefore, a concept map can be considered a PFnet in which all link weights equal one; that is, a graph. Thus, the 
length of the path between any concept map node dyad is the sum of the number of links traveled in moving from 
the initial to the terminal node. 

PCKNOT software requires an ordered list of the terminology that it will use to label nodes. In concept 
map translation, this is a list of concept names. PCKNOT also requires a file that contains the ratings of all the dyad 
pairs. PCKNOT will accept ratings of similarities between pairs or ratings of distances. This concept map translation 
method uses distances; proximity ratings. PCKNOT distances can be either symmetrical (the distance from node, to 
node 2 is equal to the distance from node 2 to node;) or asymmetrical (the distance from node; to node 2 is not equal to 
the distance from node 2 to nodej). This concept map translation methodology uses symmetrical proximities. 

Efficient and accurate implementation of concept map translations would require the use of a computer to count the 
distances between the all the network node dyads within each concept map. However, counting can be done by 
hand. When counting and translation are done by hand, record keeping is facilitated through a spreadsheet, entering 
the concept labels as vertical and horizontal matrix headings (see Figure 3). PCKNOT requires parameters for the 
shortest (min) and longest (max) allowable path lengths. When a dyad pair has not been specified within a concept 
map, it is recorded as a length that is longer than the maximum parameter. The software calculates such a dyad pair 
as infinitely far apart. 



Figure 1. Pathfinder Network Translation of the Expert Model Concept Map for the Functionalism Domain 
(Tradition Society Segment), Constructed Within PCKNOT Software. 




Participants and Data Collection 

Data for this analysis were collected as part of a larger study that investigated the effect of a metaphor- 
enhanced, computer-mediated learning environment on learners’ mental models of a targeted domain (Reese, 
2003b). Fifty-seven preservice elementary teachers studying at an R-1 university in Virginia worked through a unit 
of instruction that introduced the educational philosophy concept of functionalism. An advance organizer designed 
over a concept map of the targeted domain introduced instruction for the control group. The treatment group 
completed a unit of instruction that used metaphor-enhanced computer interface as the advance organizer, 
constructed according to the Analogical Designs Model (Reese, 2003c). All participants answered a set of four, 
timed, extended constructed response (essay) questions. A team of two trained and domain certified raters (a) parsed 
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participants protocols for evidence of 13 targeted domain concepts, (b) constructed a concept map for each response, 
and (c) collapsed each participant’s set of four maps into one concept map. Each concept map represented a 
participant’s mental model of the educational philosophy domain of functionalism. Participants’ understanding of 
content was measured by totaling subscale scores (levels, branches, and cross-links). This type of scoring measures 
the richness of a learner’s mental model. Results suggested that, while most learners can specify domain hierarchy 
(subsumption) and branches (differentiation); learners with undeveloped mental models require explicit instruction 
in domain integration. Those fifty-seven concept maps and the Expert Model concept map (see Eigure 2) upon 
which the instruction had been designed and developed served as the data for the Pathfinder analyses that follow. 

Eigure 2. Expert Model Concept Map of Eunctionalism: Traditional Society, With Labeled Relational Links. 




Creating the PFnets 

These fifty-seven concept maps and the Expert Model map were translated into PEnets by hand, using the 
Proximity Matrix (see Eigure 3). The 13 functionalism concept labels were used as vertical and horizontal matrix 
headings. The PCKNOT maximum path for this set of concepts was set as nine relational segments. When a dyad 
pair had not been specified within a concept map, it was recorded as 10 line segments (infinitely far apart). 

Eigure 4 is the concept map that was prepared by raters from protocols produced by a participant #105. 
Notice that concepts informal selection and social change were mentioned within the participant’s protocols, but 
evaluated as relationally unconnected to the other 1 1 concepts. Therefore, the two concepts were noted on the 
concept map, but unconnected. PEnets calculate only for specified relational links. When concept maps are 
translated into the Proximity Matrix, all dyad paths to the identified but relationally isolated concepts are counted as 
10, as are the dyad paths that involve non-identified concepts. Within map #105, paths involving the identified but 
unlinked concepts (informal selection and social change), as well as the unidentified concepts (stage theory, 
traditional society, informal training, and ascribed characteristics) are recorded within the matrix as 10 (infinite 
distances, see Eigure 3). 

PCKNOT reads the proximity data prepared as a simple text file. The data may be ordered as a square 
matrix, an upper triangle, lower triangle, or a list. The data file does not have to take the shape of the matrix or list, 
but data must be in the same order as they would be if in the specified shape. In this analysis, the data were entered 
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as a lower triangle matrix. To prepare this data file, data record cells within the triangle matrix in Figure 3 were read 
across, by row, from left to right. Thus, the informal education X goals proximity of “2” is the sixth proximity 
entered. The “DATA” heading within the file contains the file name (lOS.prx), the identification that “distance” 
rather than similarity ratings will be used, the number of node/concept items (13), the number of decimal places to 
be calculated, the minimum distance (0), the maximum distance (9), and the order of the data (lower triangle). For 
PCKNOT to recognize this file, it must be saved as simple text with the file extension “.prx” (proximity). PCKNOT 
reads the “prx” data file and calculates the values of the PFnet. It saves the quantified output values as a text file (see 
Table 2). This is also read by the software and projected as a network diagram (in this case, a graph, as in Figure 2). 

Analysis Through PFnets and PCKNOT 

Once PCKNOT has processed a set of concept maps, the software can calculate a number of relationships 
that afford comparisons (a) between participants and an expert model, (b) between participants’ mental models, or 
(c) between averaged models. The sections that follow will discuss the PCKNOT output files for each of the three 
comparisons. Each output presentation is followed by an interpretation and a discussion of the application of that 
type of analysis within instructional or research practice. 

Comparison to Expert Model. Table 3 displays a snippet from the PCKNOT analysis output in which 
eight PFnets (109, 113, 114, 131, 143, 148, and 155) were compared to the Expert Model The first row of data 
belongs to PEnet 105 and the last to the Expert Model, with the remaining PEnets in between. This section highlights 
aspects of the PCKNOT output. 

The first section of the PCKNOT output displays each PEnet’ s coherence, the file number assigned by the 
analysis, and the corresponding file name (see Table 3). Coherence measures the consistency of a network. 
“Coherence often correlates with expertise (or degree of learning). Very low coherence (less than 0.20 or so) may 
indicate that [participants] did not (or could not) take the rating task seriously” (McDonald et ah, 1998, p. 2). 
Networks with identical links will score a similarity of 1, and those with no links in common will score 0 
(McDonald et ak, 1998). Within this subset of PEnets, those with the highest similarity to the Expert Model are 109, 
113, and 1 14 (see Table 3, column 8). In general, the greater the number of total links contributed by each dyad 
member, the greater the possibility that the two networks will hold common links due to chance alone. The Expected 
links in Common By Chance is reported in the E[C] column (see Tables 3 and 5, column 6). Of these three PEnets, it 
is expected that 109 might have the greatest similarity to the Expert Model, due to chance alone. It is noteworthy, 
then, that (a) PEnet 108 registers a higher similarity than 109 and (b) PEnet 1 14 scores an identical similarity score, 
even though it contains only 5 links compared to the 16 links in PEnet 109. The point probability (the probability of 
exactly this number of links in common) and tail probability (the probability of at least this many links in common) 
both take this into account. The Tail probability “can be used as a statistical test of the similarity of two networks” 
(McDonald et ak, 1998). That is, a significant result means it is highly unlikely that the similarity between the two 
networks is due to chance alone. Both the 113 and 1 14 PEnets are significantly similar to the Expert Model. PEnet 
109 is not significantly similar. The PEnet 105 is borderline. 

PEnets 143 and 148 hold no relational links in common with the Expert Model. They each score similarity 
ratings of “0” and Tail probabilities of “1”, indicating that it is highly likely that the resemblance to the Expert 
Model might result as it does, due to purely chance. 

Interpreting the Results; Comparison to Expert Model. In this manner, learner and participant concept 
maps can be translated into Pathfinder networks and individually compared to an Expert Model. Within the group of 
PEnets analyzed here by similarity, post-instruction mental models held by participants 113 and 114 were the closest 
to the functionalism Expert Model. The correlation of each with the Expert Model (an aspect of the output file not 
discussed within this paper due to space limitations) further supports this conclusion. Depending upon the 
instructional goal at the time of data collection, the level of achievement demonstrated by 1 13 and 114 may or may 
not be acceptable. If the goal of instruction were for the learner to assimilate the Expert Model as scaffolding for 
further domain study, a similarity of 0.222 might be inadequate and evidence of a gap between the instructional goal 
and learner achievement, signaling a need for revision of instruction. 
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Figure 3. Proximities Matrix for Concept Map 105. 
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Figure 4. Participant 105 Concept Map. 




PFnet analyses indicate that learners 131, 143, and 148 are moving more slowly than the others in this 
group. Participants 143 and 148 have not constructed any relational connections that correspond to those within the 
Expert Model. Their PFnets contain fewer relational links than the others in the group. Results suggest that these 
learners require additional practice or remediation. On the other hand, while learner 131 has constructed a 
relationally dense model, it does not correspond to the Expert Model. This is a red flag, signaling that learner 131 
may be building a non-normative model. The instructor should examine the learner’s concept map; perhaps the 
learner has constructed an alternative model that can be substantiated. In any case, results for 131 suggest the need 
for a one-on-one interview. Follow-up may include instruction to uncover the characteristics of the misconception 
and correct it. Or, the Expert Model might require modification or inclusion of alternative versions. Thus, alternative 
insights provided by this learner might be incorporated into evaluation. Consideration of the 131 PFnet demonstrates 
how the automated functionalities of PFnet translation of learners’ concept maps can facilitate an instructor’s ability 
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to monitor learner progress and evaluate instruction, affording timely and adequate instructional interventions as 
well as revision. 

Application: Comparison to Expert Model. Results from comparison to an Expert Model can be applied 
within research and instructional settings: 

1. For research purposes, similarity ratings can be imported into a statistical package, such as SPSS, for use in 
standard statistical analyses. The data used in this discussion were drawn from an experiment in which treatment 
learners experienced an intervention and all post-instruction learners’ mental models were specified as concept 
maps. After (a) PFnet translation of both participant/learner concept maps and the expert map and (b) PCKNOT 
analysis, the similarity scores were entered into SPSS along with condition assignments. Standard statistical tests 
such as Pearson’s product moment correlations, f-tests, multiple regression, and ANOVAs were used to test the 
effect of the treatment, as well as the interaction between interface metaphors, concept maps, and learners’ cognitive 
characteristics. 

2. Instructionally, PFnets are an ideal tool for formative evaluation, feedback, grouping, and assessment. An online 
system implementing PFnet translation would allow a student to enter a concept map and receive immediate 
feedback about its degree of comparison to an Expert Model. A teacher or administrator could query the same 
system; receive immediate results for an individual, group, class, entire school, or even an entire state; and use that 
information for formative or summative evaluation of either student progress or the effectiveness of the instruction. 

Application: Comparison of Learners. The preceding section discussed PFNET comparisons between 
learners and an expert model. The PCKNOT software also affords comparison among learners. The correlations for 
this set of PFnets (not displayed due to space limitations) suggested a high similarity between the models held by 
learners 105, 109, and 155. A PCKNOT analysis was run to compare concept map 155’s PFnet to these and four 
additional learner’s concept maps, as translated into PFnets. The similarity between PFNETs 105 and 109 and PFnet 
155 were greater than would be expected due to change alone (TlProb < .05). Thus, the similarity analysis supported 
the interpretation suggested by the correlations among this set of PFnets. PFnets 105, 109, 113, and 1 14 were more 
similar to PFnet 155 than would be expected due to chance alone. However, the correlations for PFnets 113 and 1 14 
and PFnet 155 indicated little resemblance between these individual networks and PFnet 155. This means that the 
links that are not held in common within each comparison follow divergent paths. Perhaps this a minor difference 
for PFnet 1 14, in which the network is in its early development, containing 5 links with only 2 idiosyncratic 
relationships. In the case of 1 13, the learner’s mental model is a denser network of 14 relational linkages. Thus, the 
lack of correspondence may be more important in comparing 113 with the 155. The models for learners 113 and 1 14 
were both highly correlated to the Expert Model. A combination of the similarity and correlations outputs would 
help an instructor place these students into groups. The density of the mental model held by learner 113, the 
similarity of the models held by learners 105, 109, 113, and 1 14, and the alignment between learner 113 and 1 14 and 
the Expert Model might help all of these learners to refine their models of the domain as they continue to explore 
content. 



Interpreting the Results: Comparison of Learners. The similarity output suggests that these learners fall 
into two groups: (a) learners with PFnets 131, 143, and 148 and (b) learners with PFnets 105, 109, 1 13, 1 14. When 
learning activities called for group work, these results would allow the instructor to quickly assess learners’ progress 
and basing homogeneous or heterogeneous grouping decisions upon quantified evaluation of the students’ mental 
models. 



Application: Comparison of Learners. It is often desirable, and sometimes necessary, to group learners 
according to their understanding of a targeted domain. Traditionally, educators have used paper and pencil 
evaluations or informal observations to inform their decisions with respect to groupings. Computer-mediated 
assessment of mental models through PFnets provides an efficient method for comparing learners’ understanding of 
a domain’s underlying relational structure. The method is less biased than multiple choice or constructed response 
for such characteristics as writing and reading ability. Socially reticent learners are also a consideration, as it may be 
difficult to informally gauge mental models held by socially reticent learners. Concept maps allow these individuals 
the opportunity to express ideas that they might not share overtly. Here, again, since the maps are confined to 
terminology introduced within a specified domain, language skills are less biasing than they might be within 
traditional assessment instruments. So, PFnet translations of concept maps may be less biased than both informal 
and traditional assessment instruments while affording comparisons of learners’ mental models. 
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Comparison and Interpretation Through Averaged Mental Models. Each PFnet is an entity consisting 
of nodes and the quantified links that join them. Therefore, it is easy for PCKNOT to average those relational links 
across a selected set of PFnets. Of the fifty-seven concept maps translated for this set of analyses, the top quartile 
was defined as the fourteen with the highest similarity to the Expert Model. PCKNOT can compare the average high 
quartile PFNET across learners and/or to the Expert Model. Figure 5 displays the layout for the average high quartile 
model. The PFnet layout can provide useful information to the instructor, evaluator, or researcher that goes beyond 
the PCKNOT similarity and correlation results. With a few clicks of a mouse, the evaluator can gain access to an 
aggregated map for comparison to the normative model. Comparison reveals overall strengths and weakness. For 
example, comparison of this average high quartile model (see Figure 5) to the expert model (see Figures 1 and 2) 
reveals that 

1. These learners have constructed a model of functionalism in which collective forms the most subsuming concept. 
This reveals a misconception, because traditional and stage theory both subsume collective within the Expert 
Model. However, learners do recognize the role of collective with respect to the other concepts. 

2. Within the aggregate PFNET, only four of the twelve concepts connect to informal education. However, within 
the Expert Model of functionalism, informal education integrates the entire domain, forming the end-link on every 
path within the model. It is obvious from looking at the aggregate high quartile model that even the highest 
achieving learners within this sample are forming models that overlook the role of informal education. 

Application: Comparing the Average Model. As learners begin to construct mental models, comparison 
of an aggregate model to the Expert Model instantly reveals non-normative relational structures. Most importantly, it 
provides a snapshot of the collective, socially distributed model within which each individual learner would build 
individual understanding. Interventions can be instantly initiated and focused specifically on problematic areas by 
strengthening appropriate relational connections. Once again. Pathfinder analysis provides an efficient and 
illuminating diagnostic mechanism that allows evaluation of learner progress. 



Table 2. The PCKNOT Output File. 

PFNET C:\PCKNOTM05.PRX 

13 nodes 
8 links 
undirected 
12q(12 = n-l) 
infinite r 

1 minimum link weight 
1 maximum link weight 
links; 

nodel node2 Weight type 
3 6 1 X 

3 7 1 X 

4 5 1 X 

4 7 1 X 

5 7 1 X 

6 7 1 X 

7 9 1 X 

8 9 1 X 
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Table 3. Two Snippets From Output File Comparing Eight PFnets to the Expert Model. 

Coherence File# File Name 



0.994 


1 


105.PRX 


0.988 
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109.PRX 


0.955 
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113.PRX 


0.913 


4 


1 14.PRX 


0.928 
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131.PRX 


0.903 
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143.PRX 


0.998 
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148.PRX 


0.992 
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155.PRX 


0.783 
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Expert Model.PRX 



Similarities of PFnets: 
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PtPrb 


TlPrb 


Info 
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17 
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1.74 


2.26 


0.190 


0.078 


0.113 


0.529 


0.063 


0.398 
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2 


17 


16 
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3.49 


2.51 


0.222 


0.121 


0.101 


0.064 


0.089 


3.48 
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17 


12 
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2.62 


3.38 


0.26 


0.10 


0.15 


o.ol5 


0.018 


5.73 
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17 
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1.09 


2.91 


0.22 


0.054 


0.168 


0.006 


0.007 


7.12 
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17 


14 
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3.05 


-0.05 


0.107 


0.112 


-0.005 


0.277 


0.637 


0.65 
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6 


17 
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0.87 


-0.87 


0.00 


0.045 


-0.04 


0.365 1.( 


1.00 


0.00 
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17 
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0.87 


-0.87 


0.00 


0.045 


-0.04 


0.365 1.( 


1.00 


0.00 
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8 


17 


14 


4 


3.05 


0.95 


0.148 


0.112 


0.036 


0.209 


0.359 


1.48 



Table 3. Note: fll=file 1, fl2 = file2, Inl = # links in file 1, ln2 = # links in file 2, Cmn = # links files 1 and 2 hold in 
common, E[C] = expected links in common, O-EC = observed - expected links in common, Sim (similarity) = 
common links/(total links - common links), E[S] = expected similarity due to chance alone, O-ES = observed - 
expected similarity, PtPrb (Point probability) = the probability of exactly this # of links by chance, TlPrb (Tail 
probability) = probability of this number of links or more in common by chance. Info (information) = (log2)/(Tail 
Probability) 



Applications Within Educational Environments 

This discussion used a set of functionalism concept maps to review traditional concept map scoring and 
data analysis. It detailed the PEnet translation methodology and demonstrated how the PCKNOT software allows the 
instructional designer, evaluator, educator or researcher to compare PEnets according to their structural similarity 
and relational correlations. It suggested pertinent statistical analyses and illustrated how to diagnose troublesome 
content. In practice, an educator, instructional design team, or researcher would prepare an expert mental model 
(concept map) of targeted domain knowledge and translate this model into a PEnet. Students would work through 
early instructional activities and complete concept maps for the domain. A rater or computer would translate each 
student’ s concept map into a PEnet and run comparisons between students or between a student and an expert 
model. At the conclusion of instruction, the educator or researcher would repeat the process. Overall learner 
progress could be determined very easily through statistical analysis that compared overall or individual student 
results at time 1 and time 2. This discussion summarized procedures for comparing learners to an expert model, to 
each other, and for aggregating mental models across learners. It is important to realize that traditional concept map 
scoring rubrics measure the richness of learners’ mental models. They do not support empirical comparisons of 
concept maps’ relational structure. Indeed, it seems humanly impossible for an individual to eyeball a class set of 
concept maps and generalize relational structure across learners: comparison of two idiosyncratic maps containing 
13 concepts each would require an evaluator to remember and process 78 correspondences. This hardly fits within 
the Magic 7 parameter (Miller, 1956). 

The research reported within this paper used humans as translators, a costly and time-consuming process. 
Current computer-mediated technologies could afford an efficient and effortless process of information transfer and 
translation. Each learner could enter concept maps into mapping software that automatically processed the data. The 
software could translate the data to provide immediate feedback for the student. The data could also be translated 
and analyzed for evaluation, assessment, or research purposes, applicable to instructional design or research 
scenarios. As part of a suite of collaborative knowledge-sharing and curriculum development tools 
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http://teacherbridge.cs.vt.edu), the Virginia Tech Center for Human-Computer Interaction has created a basic 
concept map construction tool that can be used online by students working individually or in distributed groups. 
Although the current version of the tool has only been used to support minimally structured brainstorming and 
outlining activities, the architecture provides a foundation for more advanced capabilities such as translation to 
PFnets. In the broadest application of the tool, servers, designers, subject-matter experts, and researchers could be 
housed at central locations, such as universities and colleges. Consortia of teachers, military/government and 
industry specialists and their university partners would chart knowledge domains. The universities would maintain 
the centers for evaluation, educators would teach content and interact with learners, and learners could log onto the 
centers to upload concept maps and receive immediate feedback. 

Over a decade ago, Novak and his colleagues (Novak & Musonda, 1991) completed a twelve-year study 
that invented and used concept maps to chart the changes in 41 learners’ mental models as the students progressed 
from kindergarten through the 12* grade. Today’s technologies can enable researchers and educators to efficiently 
and effectively duplicate those accomplishments to the benefit and enhancement of learning for all students. 

Figure 5. PFnet Graph for Average Network of Top Quartile (Fourteen Networks) 
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Technology Integration Through Team Teaching in a Graduate Research 

Course 



Barbara Rosenfeld 
Carolina Mancuso 
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Abstract 

Two colleagues collaborated in a team teaching approach to infuse technology into an introductory 
graduate literacy research course. Graduate students (classroom teachers) learned to use technology such as the 
BlackBoard, distance learning platform, PowerPoint presentation software, and Internet resources. The team 
teaching approach provided the graduate students with a model for collaboration that they could apply in the 
research class and also in their own classrooms. There was a dramatic increase in the self-reported level of 
competence in the use of BlackBoard during the course, as well an increase in the self-reported level of competence 
in the use of Internet resources and PowerPoint . 

It was Monday night, midsemester, and Professors Carolina Mancuso and Barbara Rosenfeld were team 
teaching their graduate Introduction to Literacy Research course. Carolina had written directives for the students on 
the computer, and the words were projected on the screen in the front of the room. Students were asked to write 
about their writing, in this case, an analysis and critique of a research article. 

Students were often asked to reflect on their writing and the writing process. This was nothing new. What 
was new was that many students were choosing to write their reflections on the computer. This graduate literacy 
research course took place in the newly refurbished School of Education lab, where two dozen iMacs were available 
for student use. One goal of the team teaching was for Barbara to help Carolina integrate technology tools into the 
course. How did this collaboration come about? 

When Barbara started teaching at the school, she noticed that relatively few faculty members were using 
technology tools in their teaching. The Dean recognized that technology was not being used optimally to benefit 
student learning in all classes. Accreditation organizations such as the National Council for the Accreditation of 
Teachers in Education (NCATE), expect to see technology being used in higher education classrooms. With an 
NCATE review in the near future, the department began to address this deficiency. When Barbara proposed to the 
Dean that she help a willing colleague integrate technology into his/her course through a team teaching approach, 
the Dean was very supportive. The purpose of this article is to share the experience of using a team teaching 
approach to enable classroom technology integration. 

Barbara approached Carolina to ask about a possible collaboration; she was very enthusiastic. Both 
Carolina and Barbara had started teaching at the school at the same time, but didn’t know each other very well. 
Although each had expertise in her own field — Carolina in literacy, Barbara in educational technology — they also 
had had some experiences in each other’s fields. 



Background 

Collaborative efforts are fairly common in higher education, although there is an apparent shortage of 
literature related to collaboration among professors in colleges of education (George & Davis-Wiley, 2000). Austin 
and Baldwin (1992) reported that faculty collaboration has grown over the past century, negating conventional 
stereotypes of isolated professors conducting laboratory research or lecturing in a room full of passive students. 
Team teaching benefits professors by developing their pedagogical abilities, intellectually stimulating them, 
engaging them as self-directed learners, and connecting them more closely to the college community (Paulsen & 
Eeldman, 1995). 

Team teaching collaborations often involve instructors from different disciplines who merge their expertise 
in diverse ways. This collaboration paired an educational technology professor with a literacy professor. In most 
collaborations, the professors’ expertise in their content areas is combined to provide new material for the course. In 
this situation, Carolina provided the expertise for the course content, and Barbara provided the expertise for infusing 
the technology tools. 

Austin and Baldwin (1992) suggest that issues of power, influence, and professional identity can sometimes 
be problematic for collaborators, but these were not issues for Carolina and Barbara. In their guidelines for 
collaborative teaching, Cruz and Zaragoza (1998) advise of the need to establish mutual respect and trust. Carolina 
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and Barbara’s similar teaching styles and similar interests, especially their mutual love of chocolate, helped to 
quickly establish both respect and trust. 

Team teaching provides a model of collaboration for students as well (Cruz & Zaragoza, 1998; Warmkessel 
& McCade, 1997). Austin and Baldwin (1992) suggest that students should be introduced to the merits and 
processes of collaboration as part of their professional socialization. In several studies, students expressed a positive 
reaction to team teaching and felt they greatly benefited from the experience (Crossman & Behrens, 1992; Davis- 
Wiley & Cozart, 1998). In Carolina and Barbara’s class, the graduate students, themselves teachers, appreciated the 
convivial manner of the professors’ interaction that contributed to the relaxed atmosphere in the classroom. 

Implementation 

Before the beginning of the semester, Carolina had already created a course syllabus with substantial 
content. Barbara worked on ways to enhance the existing course through the use of technology tools. BlackBoard, 
an electronic software platform that contains a course management system for distance learning, was introduced to 
the students as a way to communicate both during and outside class. 

The class met in a computer lab that had 24 computer stations around the periphery of the room, and 
moveable tables and chairs in the center. This environment made it easy for students to engage in online research at 
the computers, collaborate in small groups to discuss their writing at the center tables, and then move back to the 
computers to edit their work and compose short metacognitive progress reports. Much of the class time was spent in 
this way. The two professors circulated among the students to discuss their research proposals with them. Students 
appreciated having two professors available for consultation. 

The professors modeled collaborative behavior, and the students were asked to practice collaboration in 
their writing and research activities. Students read, edited, and critiqued each other’s work. Rubrics were available 
to give students guidelines for their projects and also to ensure that they would understand the grading process. A 
long-term goal of the professors’ collaborative efforts was that the graduate students would facilitate collaborative 
activities and integrate technology tools in their own classrooms. 

The overhead projector in the lab was never used. The professors modeled technology use on a computer 
with a projection unit that was used to display the plan for the day’s session, provide initial directives, and to log 
class discussions during every session. These logs were then posted on BlackBoard, giving all students access to 
past class activities. Assignments and course documents were also posted on BlackBoard, and class or individual e- 
mails were easily dispersed. 

PowerPoint instruction was given during the semester so that those students who wished to use it to present 
their research projects could do so. Approximately one-third of the class opted to use PowerPoint for their final 
presentations. 



Team Teaching and Technology Benefits 

The two professors benefited from this collaboration. Carolina became more familiar with technology tools 
and increased her trouble shooting ability by the end of the semester. Although she had previous experience with 
BlackBoard, Barbara assisted her in learning how to use additional aspects of that program. She currently uses these 
technology tools in all her courses. Barbara learned more about the course content, particularly ways to help students 
reflect on the writing process by incorporating metacognitive exercises. A wonderful side benefit of this 
collaborative effort is that the two professors became trusted colleagues and good friends who are planning to work 
together on additional projects that will capitalize on their mutual strengths. 

Although the main purpose for team teaching was to integrate technology tools in the course, there were 
additional benefits of this classroom experience for the students who observed modeling of team teaching as well as 
technology. Students’ feedback after the course ended described their appreciation for having two professors 
available for discussion of their projects and for technology guidance. They liked receiving two opinions on their 
work, additional individual conferencing and the support that two professors could provide. 

Students also gained knowledge about classroom use of technology. The easy access to the computers in 
the classroom lab put the research tools at their fingertips and helped them complete their assignments. They 
welcomed not having to go to the library to do all their research. Instruction on PowerPoint enabled students to 
produce professional presentations. One student reported that she had some anxiety with the technology during the 
course. She worried that the technology would fail and she would appear incompetent. Since the course ended, she 
has had additional opportunities to use the technology, (in another of Carolina’s courses) and she reports feeling 
more confident. Another student tried to implement some of the things that were modeled in the class - her young 
students use PowerPoint, word processing, and the Internet to conduct research in her classroom. She also gives 
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them opportunities to collaborate in their work. By the end of the course, this graduate student had completed six 
projects in her own class by asking her young students to use available Internet resources to research their topics. 



Technology Use Survey 

A survey of competency of technology use was administered at the beginning and again at the end of the 
semester using a 4-point Likert scale (Don’t know anything, Know a little. Somewhat competent. Very competent). 
There was a dramatic increase in the self-reported level of competence in the use of BlackBoard during the course. 
At the beginning of the semester, 15 students (out of the 18 who responded to the survey) reported that they didn’t 
know anything about BlackBoard. At the end of the semester 1 1 students (out of the 17 students who responded) felt 
somewhat competent and 3 felt very competent in their use of BlackBoard. By the end of the course, no students 
reported that they didn’t know anything about BlackBoard. (Please refer to Table 1.) Questions about the use of 
automated library resources and PowerPoint also show that students felt more competent at the end of the semester 
than they did at the beginning, but the increase in self-reported competence was not as dramatic. (Please refer to 
Tables 2 and 3.) 

Table 1. Response to BlackBoard Question on the Survey of Technology Use 



What is your 
level of 
competence 
with the use of 
BlackBoard? 


Don’t know 
anything 


Know a little 


Somewhat 

competent 


Very 

competent 


No response to 
this question 


Beginning of 
semester 


15 


1 


1 


0 


1 


End of 
semester 


0 


3 


11 


3 


0 



Table 2. Response to Automated Library References Question on the Survey of Technology Use 



What is your 

level of 

competence 

with the use of 

automated 

library 

resources? 


Don’t know 
anything 


Know a little 


Somewhat 

competent 


Very 

competent 


No response to 
this question 


Beginning of 
semester 


7 


7 


2 


0 


1 


End of 
semester 


0 


3 


11 


1 


2 



Table 3. Response to PowerPoint Question on the Survey of Technology Us 



What is your 
level of 
competence 
with the use of 
PowerPoint? 


Don’t know 
anything 


Know a little 


Somewhat 

competent 


Very 

competent 


No response to 
this question 


Beginning of 
semester 


6 


10 


2 


0 


0 


End of 
semester 


3 


6 


6 


2 


0 
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Ideas for the Future and Concluding Thoughts 

There were other technology activities that were planned during the semester, but they did not transpire. 

The professors had hoped to have students try round robin editing (where they would change seats and edit each 
other’s work on the computer), but the time was unavailable to implement this technique. Another idea was to try 
editing by exchanging files on BlackBoard using the small group discussion board. This also did not occur. Another 
way that technology can help student research is by using videocameras to tape teachers and students in the 
classroom. These are all possible activities for the future. 

The administration was instrumental in promoting this innovation. The Dean supported the team teaching 
idea and permitted it to occur. Administrators have a key role to play in fostering effective collaboration by 
allocating discretionary resources and shaping supportive policies. Rewarding and recognizing collaborative 
achievements both publicly and privately can also stimulate collaborative work (Austin & Baldwin, 1992; Paulsen & 
Feldman, 1995). 

Technology integration is an important standard in many educational organizations. When faculty are asked 
to implement technology tools in their classrooms, administrators should offer the support necessary for each faculty 
member. Team teaching is an innovative way for knowledgeable faculty to help their colleagues and it should not be 
overlooked. 

One student’s final thoughts on the class provide a testimonial to its value and significance; 

The research class that I fortuitously landed in provided me with a fresh perspective on teaching, 
concrete collaborative methods of completing projects and a supportive learning environment. The 
instructors. . .truly were a dream team. They demonstrated, planned or not, how well teaching to strengths 
and being supportive to new attempts can create a strengthened instructor base, and in turn, a richer 
learning environment. 

The technology aspect of the class was complimentary to its research counterpart. It was useful to 
have the computer lab as the classroom. The convenience of access to technology, along with guided 
instruction and modeling proved invaluable to the completion of class assignments. The slide-show lessons 
were especially useful as they provided us with the means of producing a professionally appealing 
presentation.... 

In the area of team teaching, the instructors. . .allowed for increased conferencing time on the 
discussion of projects, technology guidance, and a shared therefore lighter workload (hopefully). After all, 
a happy instructor is a happy marker, creating a happy student. . . .Thanks to their fearlessness of team 
teaching, I am willing to give it a try in my own classroom. . . .Hopefully there will one day be enough 
money and compatible people for team instruction to be the norm rather than the exception. 
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Abstract 

The purpose of this case study was to examine a teacher education system relative to the degree of 
performance support for the use of technology to support learning. Degree of support was measured by the presence 
of factors such as clear expectations, feedback, tools, rewards, incentives, motivation, capacity, skills, and 
knowledge within the system. Major stakeholders within the teacher education system at the target institution 
included teacher education faculty , student-teacher candidates, and cooperating teachers in local schools. Several 
areas of performance support were found to be required within and across stakeholder groups. Results of the study 
are discussed with a particular focus on the degree of alignment of supports across the system. Recommendations 
for the allocation of system resources related to technology integration are also discussed. 

Background 

Public schools in the U.S. have been the target of reforms and federally-sponsored programs and initiatives 
almost since their inception. These initiatives focus on aspects of schools that range from the expansive, such as 
total system reform, to specific areas of focus such as faculty development, and curriculum and course management 
(need ref ?). Moreover, federally sponsored reports such as the Secretary’s Commission on Achieving Necessary 
Skills or SCANS (1991) and most recently. Partnership for 2P‘ Century Skills (2003). 

A common thread between all of these reports has been a focus on what students should know and be able 
to do upon graduation. SCANS and Workforce 2000 (Johnston & Packer, 1987) emphasized learning to learn, 
problem solving with visual tools such as graphs and charts, and working collaboratively on teams. The Partnership 
for 2P‘ Century Skills report included digital literacy and the use of technologies for the purpose of communicating 
and collaborating. Another recent technology-oriented focus of school reform at the federal level has been in the 
area of integrating technology into classrooms through a variety of programs, such as the Technology Literacy 
Challenge Fund and Preparing Tomorrow’s Teachers for Technology (PT3). 

Given these and other reform efforts at the state and school district level, why has the K-12 educational 
system not been successful in integrating technology in order to provide students with those necessary skills? 
Perhaps a look at the progression of professional development efforts related to technology integration will help to 
answer this question. These efforts got under way in earnest in the early 1990's. At that point in time, funding and 
related activity concentrated on getting the necessary hardware into the schools. The overarching problem in this 
phase was that teachers were not being trained on how to use the equipment. The mid-1990's, or phase two, saw a 
change in the approach of integrating technology into our schools. Many of the new funding programs, including the 
Technology Innovation Challenge Grant Program and the Technology Literacy Challenge Fund, stipulated that 
professional development should account for a major portion, at least 30%, of the funding while the remainder was 
used for acquiring equipment. This was done to ensure that educators would have the ability to use newly acquired 
computer equipment so it would not sit dormant in the classrooms and schools as in the previous phase. In other 
words, educators were taught the necessary skills, leading them on a path to become technology literate. 

We have now reached a third phase of technology integration reform movement, in which educators have 
either decided it is not worth the effort to integrate technology or they have adopted technology. For those who have 
adopted technology, many are not truly integrating technology but rather using it on a superficial level (Becker, 
2001; Education Week, Technology Counts 2003), as is allowed by their skills, and the fact that they are technology 
literate but not yet “technology fluent’’. These educators have reached a point where they are ready for further 
training, information, and most importantly, knowledge that incorporates pedagogically-sound principles for 
integrating technology correctly, efficiently, and effectively. 

During this phase of the reform movement to integrate technology it is time to look at the situation from a 
broader perspective. Instead of trying to go for the “quick fix’’ approach for perceived problems at the K-12 level. 
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perhaps we need to look at the larger educational system, including pre-service teacher candidates and the 
curriculum they experience, as well as those that instruct them, the teacher education faculty at colleges and 
universities. 

Several recent reports (Smerdon, et ah, 2000; Moursand and Bielfeldt, 1999; Panel on Educational 
Technology, 1997; Office of Technology Assessment, 1995) have looked at the use of technology by faculty in 
teacher preparation programs. Findings related to their use of technology has shown “that technology is not central 
to teacher preparation in most colleges of education. Problems include limited use of technology in teacher 
education courses, an emphasis on teaching about technology rather than teaching with technology, lack of faculty 
modeling, insufficient funding and faculty professional development opportunities, and lack of emphasis’ on 
technology in students’ field experiences (Lehman & Richardson, 2003)”. 

While the development of skills to use technology is necessary to supporting its use, it is not sufficient. 
Faculty members in the teacher education system and teachers within K-12 systems have historically been 
introduced to new technologies using a faculty development model. 

This approach assumes that skill development training will produce the desired result, use of technology in 
the classroom, but ignores non-training issues such as incentives (Education Week, 2003). 

To address the development and resource requirements of the faculty in a teacher education program, a 
performance support approach has been found to be appropriate (Wedman & Diggs, 2001). In this study at the 
University of Missouri, faculty within the teacher education program were surveyed regarding perceived barriers to 
technology use. Several barriers were identified including unclear expectations for the use of technology, inadequate 
tools and equipment, lack of incentives promoting the use of technology, and lack of knowledge about how to use 
technology. 

A performance support approach implies that multiple elements within a performers work environment 
combine to support excellent performance. These elements can serve as enhancers of performance as well as 
barriers. Performance support system models have been used to analyze problems and opportunities in organizations 
for many years (Douglas & Schaffer, 2003). Models have been developed to study workplace behaviors and 
management of contingencies in the environment (Gilbert, 1968), as well as the horizontal and vertical 
interrelationships between people and processes across organizational functions (Rummler & Brache, 1995). There 
are many similar models described in the human resources, organizational development, and performance 
improvement literature (see Schaffer, 2000 for a review of models). 

In the current study, the performance pyramid developed by Wedman & Graham (1998) was used to 
identify the presence of support for the use of technology in K-20 classrooms by various stakeholders (see Figure 1). 
The performance pyramid is composed of eight elements that are believed to significantly impact the ability of a 
performer to produce valued results in the workplace (or classroom). These elements include: 

• Vision - An organization’s stated mission, goals, and strategies related to achieving superior results and 
being a value-added member of society. A critical aspect of creating an organizational vision is the 
alignment of mission, goals and strategies throughout the organizational structure. For example, a 
university strategic plan is set by the president and board of trustees. This plan is then cascaded throughout 
the organization. Colleges, schools, depts., programs, faculty, and professional staff are then expected to set 
their own strategies that are aligned with the university strategic plan. 

• Culture - As the organization attempts to carry out its vision, the members of the organization codify 
knowledge, rules of engagement, modes of communication, norms, approaches to problem solving, etc. 
which all add up to its culture. 

• Expectations & Feedback - To perform a task or achieve a goal, it is necessary that performers know 
precisely what is expected in order for the task or goal to be achieved. Expectations may be a set of 
guidelines, a specified level of performance measured quantitatively or qualitatively. This measurement of 
performance is then fed back to performers to let them know how well they are doing relative to the goal. 
Without feedback, performers have little information about their current performance that will help them 
improve. Setting expectations and providing a feedback mechanism is primarily the responsibility of 
leadership and/or management. 

• Tools, Information, Processes - Critical tools, information and processes related to performing tasks must 
be in place to ensure that performance is optimized and performance expectations can be met. Performers 
may have knowledge and skills about how to perform but without up-to-date tools and information or clear 
processes it will be difficult to perform to expectations. Making sure tools information and processes are 
available to performers is primarily the responsibility of leadership and/or management. 

• Rewards & Incentives - Monetary and non-monetary rewards and incentives for performance of critical 
tasks will greatly enhance the likelihood of improved performance. Performers in organizations expect to 
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be compensated fairly for contributions to the accomplishment of organizational mission and goals. 
Rewards and incentive systems are closely linked to individual motivation. Putting rewards and incentive 
systems in place is primarily the responsibility of leadership and/or management. 

• Desire and Self-concept - Desire is the extent to which performers buy into the vision, mission, goals, 
expectations and rewards structures as evidenced by their desire to perform. In essence, desire to perform is 
the “consequence of the values, background and work habits an individual brings to a situation” (Wedman, 
2003). Self-concept relates to one’s ability to see themselves performing a task. This ability can be 
enhanced by the presence of role models and by putting performers in situations in which performance is 
enabled. 

• Capacity to Perform - Humans have different capacities to perform roles or tasks and roles or tasks require 
specific capacities. Physical and mental capacities along with cognitive style required by a task must be 
matched with the individual. For example, designing web pages requires the capacity to visually 
discriminate on-screen icons. This task may also require the capacity to sit for long periods at a time. 
Accommodation may be made to work around capacity limitations but accommodations may also require 
the presence of other types of performance support such as skills or tools. 

• Knowledge & Skills - Different tasks require varying degrees of knowledge and skills. Knowledge and 
skills are what people should know and do to be able to perform a task. Education is often thought as 
supplying knowledge or ability to solve novel problems, and training is geared toward development of 
skills related to a specific task but these concepts tend to be mitigated by experiences, background and prior 
knowledge performers bring to a given situation. 



Significant Accomplishment 
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Figure 1. Performance Pyramid (Wedman & Graham, 1998) 

Research Questions 

To more fully understand the complex issues surrounding integration of technology into classrooms, a 
teacher education system at a large Midwestern university was studied. The following research questions were 
examined: What performance support factors have the greatest impact on technology use by teacher education 
system stakeholders? What factors limit or serve as barriers to effective use of technology in the K-12 classroom? 
What factors in a teacher education program limit or serve as barriers to effective preparation for use of technology 
to enhance learning? 

To study barriers to technology use in the classroom, we will partially replicate and build upon a study 
done by Wedman & Diggs (2001). Their study focused on barriers to technology use within the teacher education 
program at the University of Missouri. While they studied barriers for teacher education faculty, this study also 
examines teacher education students (pre-service) and in-service teachers in area K-12 classrooms. 
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Method 



Stakeholders 

Teacher education faculty in the Department of Curriculum & Instruction within the School of Education, 
students in teacher education classes were participants, as were cooperating teachers at area K-12 schools. Twenty- 
three faculty members out of the 46 within the department completed the survey, and 6 members of the faculty were 
interviewed. A total of 21 students completed a questionnaire and a total of 6 were interviewed. Twenty-two 
cooperating teachers (CT) completed the questionnaire and 1 was interviewed for the study. All participants 
volunteered for the study. 

Survey 

Questionnaire and interview items were based on the Performance Pyramid elements (see figure 1) and 
adapted from Wedman and Diggs (2001) Technology and Leadership System Survey and the Technology 
Leadership Support System Interview for teacher education program faculty. Specifically, companion survey 
instruments were developed for student teachers and cooperating teachers. Additionally, interviews included items 
related to ‘culture” of the organization as it relates to support of technology use. Questionnaires were administered 
on the World Wide Web thus it is difficult to calculate an exact return rate. Laculty respondents represented 50% of 
the teacher education faculty, student teachers represented approximately 15% of the active student teachers in the 
semester data was collected, and participating cooperating teachers represented 23 schools across the state that 
partner with the university to train student teachers. 



Findings 

Questionnaire data collected for each of the stakeholder groups is shown in Table 1. The percentage of 
respondents within each stakeholder group indicating the presence of each type of performance support is shown. 
The percentages that are bolded are thought to indicate a particularly low level of performance support for the 
stakeholder group. Across the teacher education system, performance support elements that were most lacking 
include: 

• Rewards and incentives for the use of technology was the most significant barrier for pre-service and in- 
service teachers; university faculty also saw this as one of their major barriers. 

• The presence of feedback regarding the appropriateness of current technology use was a big outage for 
faculty and students in the university. In-service teachers also rated this as one of their most significant 
barriers. 

• Expectations for technology use in the classroom was a barrier for faculty and pre-service teachers within 
the university. 

• The degree to which the physical environment within classrooms and labs supported technology use was a 
barrier for faculty and pre-service teachers. 



Pyramid Block 


CT n=22 


FAC n=23 


STU n=21 


Skills & Knowledge 


76 


53 


81 


Capacity 


71 


88 


76 


Motivation 


86 


88 


71 


Rewards & Incentives 


5 


28 


9 


Access/Phys. Env. 


95/71 


65/56 


95/52 


Process support 


62 


72 


33 


Eeedback 


33 


18 


10 


Expectations 


62 


41 


28 



Table 1. Presence of performance support for use of technology across stakeholder groups. 
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Faculty interview data supported the survey findings. For example, the faculty interviewed (n=6) all 
pointed to the PT3 grant at the university as an opportunity that provided not only training in skills and integration, 
but also as offering incentives and reward. However, this was seen as a temporary opportunity, and several 
mentioned that they were unsure of what would happen when the grant ended, with at least 2 predicting things 
would return to the “status quo”. In terms of feedback and expectations, the faculty discussed that while technology 
was not a requirement it was encouraged and valued, with no detriment to their careers if technology was not 
employed within the curriculum. Several also indicated that while feedback was informal they did perceive 
themselves as receiving recognition for their use of technology. 

In relation to organizational culture and vision, the faculty indicated that they felt there was a strong value 
associated with technology at the university level, in accordance with the university’s strategic plan developed over 
the past year. They also indicated that at the departmental level there was a strong vision, mostly associated with the 
Department Head, who encourages and values technology. However, it was also mentioned that their had been an 
interim Dean in the School until the current academic year, thereby putting on hold the alignment of the School of 
Education’s strategic plan with that of the university’s. 

The student interviews (n=6) reflected a different picture. In terms of the culture in the teacher education 
program, responses reflected a continuum of experiences; from “no technology is expected or exists” to “technology 
is always utilized”. In terms of the vision, they indicated that basic skills were encouraged and/or a comfort with 
technology, but not much beyond that. In relation to the survey questions, the student interview data supported the 
survey findings, reaffirming their need for support related to technology integration and skills (skills and 
knowledge), a lack of rewards or incentives, a need for feedback and a better understanding of what is expected of 
them in terms of technology use. 

Open-ended Survey Questions 

At the end of the surveys given to the cooperating teachers, the pre-service teachers, and the faculty, each 
was asked what they about the roles o the other groups in relation to preparing the pre-service teachers to integrate 
technology. For example, the cooperating teachers were asked what role, if any, they thought they should play to 
help pre-service teachers use technology in the classroom. While the responses ranged, most (n=18) indicated that 
they thought they should play an active role through modeling, discussing issues, and developing lessons. As one 
cooperating teacher stated: 

I think it’s important to help student teachers incorporate a variety of technology 

options during the student teaching experience. As she observes me using technology, 

she gains more knowledge and confidence in using technology on her own. 

On the other hand, some of the cooperating teachers also mentioned that the pre-service students sometimes 
know more about technology than the teachers, leaving them out of the technology loop. As one cooperating teacher 
explained, “Since I do not feel comfortable with technology, I do not feel I would be beneficial in demonstrating 
this.” Moreover, the cooperating teachers indicated that while they should play a role, teaching the pre-service 
teachers how to use technology was not among them; instead, they feel that is the responsibility of the faculty. 

When the pre-service teachers were queried on the same topic, what role their cooperating teachers should 
play, almost all (n=19) all indicated an informative role for the cooperating teachers , such as modeling technology 
integration, discussing what has worked for them in their classroom, discussing when to use it and how to use it 
effectively. Also, as expected, several (n=4) indicated that the technology resources were so minimal that their 
cooperating teachers couldn’t really help them in this regard. 

In response to the role of the faculty in the teacher education preparation program at the university, the pre- 
service teachers were split, with approximately 50% saying that they learned many skills and techniques in their 
teacher education program. However, the majority of those students also cited the required educational technology 
course as the basis of their experience with integrating technology. As one student shared, “I wasn’t aware of faculty 
helping me use technology other than when I took [the required educational technology] course.” Similarly, another 
student replied “They should have integrated technology into our classes if we are required to integrate it into our 
lessons.” As several of the students went on to explain, the required educational technology class and the parallel 
courses that modeled technology occurred during their first year of teacher preparation, and then technology 
disappeared for the most part, leaving them frustrated. 

Interestingly enough, when faculty were asked about the possible role of the cooperating teachers in 
relation to technology integration and pre-service teachers, many of the faculty gave responses indicating that 
technology wasn’t readily available in the schools, which was counter to what many of the cooperating teachers 
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indicated. Overall, however, faculty appeared to mirror the same expectations as the pre-service teachers in terms of 
responsibilities of the cooperating teachers, including modeling behaviors and allowing for opportunities. Finally, it 
is also important to mention that several faculty (n=4) indicated that they would need to have a better idea of what 
occurred in the schools before they could respond to this question. These discrepancies could indicate a breakdown 
in the educational system in terms of expectations, and even preparation. For example, one faculty member stated, 

“. . . we need to find out what currently exists within their schools and how we can build upon it. It is very possible 
they are doing great things and we, faculty, could learn from them.” 

Conclusions and Recommendations 

Several barriers to technology use in a teacher education system were identified during this study. The first 
research question in this study focused on the whole system and barriers common to all stakeholders. Major barriers 
were a perceived lack of rewards and incentives, and insufficient feedback regarding the level and type of 
technology use relative to expectations. On the surface it is easy to dismiss rewards and incentives as a typical 
complaint within underfunded educational systems. However, the combination of insufficient feedback and lack of 
reward and incentives suggests a more fundamental problem. The lack of feedback related to technology use implies 
that expectations for it use are not clearly understood. It is difficult to reward performance that is not measurable or 
where the results are not clearly observable. Stakeholders across the teacher education system in this study appeared 
to have a relatively strong desire to use technology in the classroom and were receiving training on the use of tools, 
but many still felt unsure about how much integration they were to do and how often they were to do it. Without a 
strong mandate from leadership, i.e. principals, department head, deans, presidents, a “volunteer” level of 
technology integration can be expected. 

Several cooperating teachers in the K-12 schools expressed a sense of ambivalence about the degree to 
which they were responsible for helping pre-service teachers integrate technology. From a performance support 
point of view, this ambivalence could be rooted in both organizational and personal issues. The degree to which a 
district and principal put forth a vision that includes technology integration and has a specific role, expectations, and 
goals for teachers in that integration helps explain the level of buy-in to such integration by teachers. A given 
teacher’ s level of skill or knowledge related to technology and a related desire to engage in technology integration 
activities helps to explain the level of resistance to assisting student teachers in the classroom. 

Faculty with the role of preparing student teachers have similar organization and individual challenges. The 
degree to which the university, school, and department visions are aligned with respect to technology will to a large 
extent explain how effective local leaders are in making expectations for technology use clear, developing goals that 
allow for meaningful feedback regarding progress, and rewarding performance that appears to support the 
organizational vision. Faculty had a sense of the organizational vision as it related to technology use but did not see 
a strong connection between the vision and performance expectations and rewards structures. In effect, there was 
little relationship between tenure and promotion and technology integration in the minds of faculty. 

Student teachers are stakeholders and customers within this system. The organizational level supports they 
are dependent upon for success are the same supports that the faculty and cooperating teachers depend upon. Thus 
students’ success at integrating technology is quite interrelated with the success their university and cooperating 
teachers have had within their respective institutions. The teacher education classes they attend collectively serve as 
models for their future classrooms. Students in this study were not sure what level of technology use is expected and 
when they are doing what is expected. The lack of feedback related to level and appropriateness of technology use 
and related lack of rewards was thus completely misaligned across the system. Students had a unique barrier in the 
lack of process support for technology use. Other stakeholders had access to a technical support person who had the 
job of making sure teachers technology requirements were supported. Students have no technical support related to 
technology integration and are somewhat dependent upon teachers or experts outside of the teacher education 
system to support them from a technical point of view. 

The results of this study show the importance of thinking systemically and supporting performance across 
stakeholder groups. Support of technology integration in schools too often focuses on acquisition of hardware and 
software, and associated skills training. It is recommended that a broader definition of technology integration be 
considered that goes beyond digital literacy to digital fluency with a focus on learning to learn with technology 
tools. From a strategic resource allocation point of view, performance support models help to elucidate specific 
ways to target resources in order to optimize technology integration. 
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Abstract 

We presume that models and theory in instructional design inform professional practice, but theory has not 
been consistently built from the professional experiences of instructional designers. This study draws on the 
observations of five instructional designers who discuss their professional identities, their communities of practice 
and their roles as agents of social and institutional change. This study is embedded in two theoretical positions: 
instructional design as a social construct that is expressed in professional communities of practice and critical 
pedagogy, in which designers act as agents of social change 

Introduction 

Instructional design is richly informed by theory, and much of the theory is practical and applied. 
Instructional designers have been trained in the theoretical underpinnings of their practice, and they use theory to 
inform their work. But much of the extensive work describing theoretical models of instructional design (ID) has not 
been drawn from the practice of the instructional designer and, consequently, instructional design theory is not 
grounded in practice. It is important to draw on the professional experience of instructional designers, their personal 
understanding of and values related to learning with technology, and the relation of these to their practice. 

This is the first in a series of studies devoted to understanding roles played by instructional designers as agents of 
social change in higher education. This study is a preliminary investigation of the professional identities of 
instructional designers and their communities of practice. Accordingly, the study had three principal objectives, 
stated here as key questions: 

• How do instructional designers describe their professional identities; that is, where do 
instructional designers draw their identities? 

• Where do designers find communities of practice, and how do instructional designers 
participate in communities of practice? 

• How do instructional designers describe their roles as agents of social change and 
transformation? 



Context Informing the Study 

This study is embedded in two theoretical positions: instructional design as a social construct that is 
expressed in professional communities of practice and critical pedagogy, in which designers act as agents of social 
change. A cultural shift has been occurring over the past decade in education and instructional design - a shift 
towards environments and approaches based on the ideas of social constructivism. In this worldview, learning is 
situated in rich contexts, and knowledge is constructed in communities of practice through social interactions. Cobb 
(1996) argues that knowledge is not held objectively, but is unique, wholly subjective, and passed on by establishing 
common ground between the knower and the learner. This common ground must embrace interests and personal 
values, which requires a sharing at both the socio-cultural and the cognitive levels (Ewing, Dowling, & Coutts, 

1998, p. 10). Constructivists are interested in prior experience, but prior experience that is shared, through 
conversation, negotiation, and construction of new knowledge products. In other words, an individual’s (designer’s) 
practice, to which self-reflection is critical, will reflect his or her values and belief structures, understandings, prior 
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experiences, construction of new knowledge through social interaction and negotiation within professional 
communities of practice. 

Communities of Practice 

The theoretical grounding for this study resides in the literature of communities of practice. We suggest that 
understanding communities of practice is central to the notion of a social constructivist view of the practice of 
instructional design. There is a rich and growing theoretical literature that explores essential characteristics of 
communities of practice (Alani, Dasmahapatra, O'Hara, & Shadbolt, 2003; Palloff & Pratt, 1999; Schwen, Kalman, 
Hara, & Kisling, 1998; Schwier, 2001; Wenger, 1998). We presume instructional designers use communities of 
practice to learn, question, critically analyze, reflect on, and negotiate their understanding of this complex field of 
study. At the same time, we know very little about instructional designers’ communities of practice. 

Social Change Agency and ID 

We suggest that instructional designers are actually engaging in a process of professional and personal 
transformation that has the potential to transform the institution. Rogoff (1990) argues that participation in learning 
hinges on communication between people in a group, in terms of shared understanding or shared thinking. Glaser 
(1991), Ewing et. al. (1998), and others (cf. Jonassen., Dyer, Peters, Robinson, Harvey, King, & Loughner, 1997; 
Kanuka & Anderson, 1998) believe that learning is most effective if it is embedded in social experience, and if it is 
situated in authentic problem-solving contexts entailing cognitive demands relevant for coping with real life 
situations, and occurs through social intercourse. The instructional design process, in which designers and others 
develop new ideas and understandings through conversation, may be a form of cultural learning or collaborative 
learning. These collaborations, however formed, may express themselves as communities of practice. 

Research Protocol 

This study was constructed as a disciplined inquiry using a grounded theory approach, and the research 
protocol was designed to actively engage designers in the research process. Sources of data included interviews with 
instructional designers, email, and group meetings and/or focus groups. Primary data were gathered though semi- 
structured interviews with five individual instructional designers from four different organizations, each of which 
lasted from one to three hours in duration, follow-up conversations using email, and from a focus group of eight 
instructional designers at one organization that lasted approximately one hour. One participant was unavailable for a 
face-to-face interview, so that interview was conducted via email. Initial interview questions were developed based 
on variables drawn from literature on communities of practice and knowledge management. Additional questions 
were developed using the language of participants to add detail and clarify their meanings. 

Results and Discussion 

The research questions defined three areas of investigation, and they were used to help organize semantic 
networks from the data. Themes emerged from the data, and they created three large clusters for analysis: identity 
and instructional design, instructional design communities of practice, and social change agency of instructional 
designers. The results are listed below without elaboration, given the limitations of the space available for this 
report. References to supporting data from coded interviews are listed by participant pseudonym and coded item, 
such that (Deborah:54) refers to the 54* coded item in Deborah’s transcript. 

Identity and Instructional Design 

Identity is an important part of any community of practice. It embraces a sense of shared purpose. A 
successful community needs to have boundaries that define its recognized focus. Sometimes the moniker 
"instructional designer" is adopted by an organization before that identity is defined, as organizations create the 
positions and anoint employees with the label. We speculated that people create identities from their experience and 
background, and in professional communities they draw on institutional culture, professional literature, professional 
organizations and reflection to understand the boundaries of their practice. 

The participants made several comments that suggested that ID is an area struggling for identity, a positive 
image and respect. First of all, there is the issue of whether ID is objectivist and hard-nosed. Add to this the 
expressed concern that ID is expensive and a process that actually slows down organizational efforts to produce 
resources (Deborah:63; Deborah:52; Eric:41) and worry from instructional designers about the importance or 
consistency of their contributions (Allen:72; Allen:80; Barbara:45). Does this mean that IDers don't understand the 
types of contributions they make? No. Instructional designers were able to identify places where they had a personal 
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impact on social change (Deborah;54), or where they brought particular skills or strategies to bear on projects 
(Eric:37; Allen:82; Allen;83). But instructional designers also recognize that their worth may be misunderstood, and 
that they will need to be able to respond to challenges from clients about their value. 

There seem to be several paths people can take into the profession, only one of which is through 
professional programs, and this may contribute to the disquiet about identity. An examination of the quotations 
coded to "experience" reveals that experience is a path to training and the profession as often as it happens the other 
way around. ID is a profession that doesn't require credentials to practice, and there is no impediment to gaining 
experience. People come to the profession from many different walks of life, and are "hooked" by the experience of 
designing something successfully. This often leads them to pursue professional training to understand the processes 
underlying successful performance. 

Teaching background seems to be particularly influential. Several participants commented on the 
importance of teaching background to the practice of ID — that it lends a valuable perspective to the practice of ID. 
There is a hint that a teaching background, and perhaps other careers, are super-ordinate to ID theory or experience, 
that teaching background not only informs the practice of ID, it is somehow more important 

It is also interesting to note the tone of the statement "...it wasn't just some instructional design theory, and 
it wasn't just based on instructional design experience." The ID identity is wrapped up in value statements about 
previous positions or careers. In fact, other instructional designers on the team were identified by their earlier roles, 
and there is the implication that these positions impinged importantly on their identities as instructional designers. 
But it is just as significant that teaching experience alone wasn't seen as adequate preparation for instructional 
design. Other experiences — as a student, in learning theory, with experienced designers — are all seen as contributing 
to ID performance (Barbara: 18). 

Institutional Culture 

Part of instructional designer’s identity is embedded in the context of the institutional culture in which ID is 
practiced. The culture of an institution carries with it very strong embedded values and a unique identity. Quite by 
accident in this study, three of the participants worked in university settings and three had worked or still worked in 
organizations devoted to designing instructional courses and resources for open learning. One person had recently 
left an open learning organization and moved to a university setting. It was quite apparent that the cultures of the 
organizations were influential, and that members of the organizations thought a great deal about the implications of 
those cultures on the practice of instructional design. (Barbara:5). 

How do the cultures differ? For one thing, the open learning organizations seem to pay much closer 
attention to the audiences, perhaps because they are viewed as customers of their products. (Barbara: 10) By 
comparison, university settings seem to emphasize content in design. (Barbara: 12) These instructional designers also 
reported finding a more business-like, project-centred ethic at work in the open learning settings. (Barbara:32) This 
was evidenced in comments about the close attention paid to project deadlines, the union mentality about work 
hours, and the efficiency of housing production and instructional support services in the same building as 
instructional design. 

University settings were viewed quite differently. Services were seen as more distributed and separated at 
the universities described by these participants. University culture was viewed as no less demanding, but the 
atmosphere and expectations were quite different. Universities seemed to adopt a more contemplative, scholarly 
approach to instructional design, and expectations emphasized research and best practice. (Barbara:33; Eric:57) 
Reading these accounts, it is possible to conclude that university life is leisurely and casual, whereas open learning 
settings are more corporate and demanding. This would me misleading. The workload demands were seen as 
roughly equivalent, but the time commitment was interpreted differently in the two settings. Open 
learning organizations ran on strict workday schedules (Barbara:34), whereas universities had little concern or 
respect for the difference between personal and professional time. (Barbara:37) But the most important conclusion 
to be drawn from this is not the direct comparison of a business model and university model of ID cultures. It is 
appropriate to reiterate that these observations are highly contextual, as are all of the observations in this report. 
There is no single university structure, nor is there any single business culture that can be generalized. Nevertheless, 
the comments harvested here point out very eloquently that different ID cultures do exist in various organizations, 
and these cultures influence the communities of practice. 

ID Professional Organizations, Literature and Reflection 

Professional organizations play an important role in nourishing ID communities of practice, but it appears 
to be an indirect relationship. By sponsoring conferences, contribute to designers' communities of practice. When 
asked about professional organizations and conferences, participants indicated that they were important for making 
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personal contact with individuals, often for the purpose of making virtual contact with them later. But as important 
as face to face contact is, conference attendance is sporadic due to the high cost of travel and attendance. And to 
foreshadow an observation we will elaborate later, professional organizations are formal structures, whereas ID 
communities of practice are largely informal. 

When asked about ID literature reviewed regularly, participants mentioned web-based resources and 
references. One mentioned two professional journals, but it is apparent that these designers more often look to the 
web for professional guidance. Formal, printed literature held no particular prominence in the immediate attention of 
these designers as a more legitimate source of trustworthy advice than web-based resources. But these designers use 
professional literature and resources, often to reflect on projects underway. 

Reflection informs the professional identities of instructional designers, and consequently their 
communities of practice. Although nobody mentioned using a formal approach to reflection such as journaling, most 
suggested that they engaged informally and regularly in professional reflection. There was also mention of group 
events and professional development opportunities that stimulated reflection. 

In the final analysis, instructional designers seem to have a clear idea of who they are and a great deal of 
respect for what they can contribute. That clarity and respect is not shared by the organizations they serve. 
Instructional designers come from many professional walks of life, and there doesn’t seem to be a central rallying 
post for them. They appear to be very competent people caught up in doing a good job. They attend conferences, 
read literature (mostly on the web), and reflect informally on their performance and how to improve it. But the 
focus is close to the ground, for the most part, not in the clouds. 

Instructional Design Communities of Practice 

Collaboration with other designers is a key element in developing a community of practice, and before 
starting this research, we assumed that instructional designers would be working in very solitary, very isolated 
settings. We were wrong. Collaboration is identified as important to instructional designers (Allen;17; Barbara:36) 
for a number of reasons and they actively seek convenient opportunities to engage others. It expands the pool of 
experience on which designers draw, and there is evidence that people include collaboration as part of the social 
fabric of the organization. 

Communities of Convenience 

It is quite apparent that Communities of Practice, as defined by participants, were born of convenience 
more than through formal structures, encouragements or agendas. Every participant in the study indicated that their 
communities were informal and convenience seemed to be a core issue in their formation. 

They were also often tied to social gatherings as predictable as a coffee klatch (Barbara:36) or as inventive 
as an exercise group (Deborah;42). The notion of convenience was central to participation in virtual communities of 
practice too. (Eric:29) They are fit into spare time, and participants mention that they design the way they participate 
in online communities so that the material is convenient to use. When communities are inconvenient to use, they are 
seen as being of lesser value. (Allen:40). In most cases, participants mentioned that other members of their groups 
were called on to advise on specific project challenges or issues (Deborah:37), but there was also mention of 
discussions of larger business, cultural, pedagogical or theoretical issues (Barbara:53) 

Physical proximity of people within an organization influences the membership in an informal community 
of practice, probably because it supports the convenience factor. If people sit near each other, they naturally interact. 
This informal engagement leads to shared experience and the development of a shared culture. 

Physical proximity also draws artificial boundaries around the community, and it excludes some others who 
work in other areas. This, of course, is no surprise; management literature is rife with advice about where to place 
employees in workgroups. But it is not necessarily considered by managers of instructional design. It may be that 
adjusting the physical seating of individuals in ID workgroups may be one of the single most important 
contributions to promoting the development of a community of practice. How does this translate to online 
communities? Virtual proximity may include how immediate or apparent the community is in the routine 
workspace of the user. Eor example, a list that pushes email to the user, or a chat space that is always available and 
alerts the participant when someone logs on may create a proximal virtual environment for the participant. 

Trajectories of Participation 

Trajectory of participation is a phrase coined by Wenger (1998) to describe the direction that people in 
communities of practice are moving. Are they orbiting the periphery of a community? Do they occupy a medial or 
peripheral position? 
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The amount of time one is in a community of practice mediates the position in the community. A new 
person in a community is naturally on the periphery, given that the designer is struggling to learn the cultural 
conventions that are second nature to more experienced members of the community. This is evident, even when a 
very experienced designer joins a new community of practice. This suggests that communities of practice are 
localized and contextual; one can't transport experiences from one community of practice to another, especially 
when the embedded cultural values of the communities differ substantially. 

There is a sense that it is necessary to get experience with a particular community of practice in order to be 
accepted into its inner circles. At least, that is the perception of some participants — that they are isolated when they 
are new to a community, but they will move to a more central position once they gain experience with 
the community. (Eric:59) . 

Sometimes, an individual’s position in a community is defined by the formal role or position held in an 
organization. Dynamic communities of practice don't automatically accept them as leaders or participants, but 
ultimately, they require social acceptance in order to perform effectively in the community of practice (Eric:60). 

Features of ID Communities 

History: Communities are supposedly stronger when they share history and culture. Conversely they are 
weak when they are based on general interests and abstract ideas. If this is the case, then the communities of practice 
within instructional design would seem to be characteristically weakened by their lack of a sense of shared history. 
There were no explicit comments about the shared history of instructional designers within a particular community 
of practice, with the exception of a comment about how the military roots of systems models of instructional design 
contribute to suspicions by people outside our communities. 

But this doesn't mean that the communities don't share a history; it is just that the participants don't think to 
talk about it as a prominent feature of their communities. The fact that most communities of practice identified by 
participants were in-house communities may mean that the shared history is so rich or intimate that it is taken for 
granted. 



Participation: Participation, even passive participation as a spectator, was seen as a key element 
of community. It can be interpreted as the frequency of contact with other members of the community. This feature 
was common to face to face and virtual communities. In virtual environments, evidence of participation included 
the number of messages, their length and whether conversations continued. This is consistent with the notions of 
engagement and alignment described by Schwier (2001). Members of communities of practice go beyond interaction 
and engage each other in meaningful conversations. This engagement, over time, leads participants to align 
themselves with the community and its central values and messages. 

Mutuality: Participants mentioned that mutuality is an important feature of participating fully in a 
community of practice. Communities spring from, and are maintained by interdependence and reciprocity. 
Participants typically construct purposes, intentions and the protocol for interaction. But mutuality appears to be 
something that is "grown into" by participants. One spoke to the notion of growing into a position where it 
was possible to make contributions that were as valuable as the ones received by the community. 

Another participant mentioned that experience with the group was just as important as having a fresh 
perspective or unique knowledge to share. New members of communities may have ideas that are useful, but they 
expressed concern that established communities may be reluctant to welcome them (Barbara:24). 

Plurality: Communities draw much of their vitality from "intermediate associations" such as 
expert/authoritative contacts, and other groups in areas peripherally related to instructional design. Participants 
spoke frequently and passionately about the associations they had with groups outside their organizations and 
immediate instructional design roles, and how those groups informed their ID work. 

But plurality is also a feature within some organizations. It was clear that some organizational 
cultures encouraged, and even mandated, that various departments within the organization that had influence on the 
learner's experience, broadly defined, were brought into the instructional designers' teams, and by extension, 
their communities of practice. 

Plurality appears to have importance, not just for the particular learning challenges being faced, but also for 
the institutional culture. When plurality has strong influence in an instructional design community of practice, 
it means that instructional designers will be bringing a wide range of considerations and solutions to bear on 
learning problems. When complex learning systems are designed in a culture that encourages plurality, there is a 
higher likelihood that all important variables will be considered. 
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Tacit knowledge: As reviewed earlier, knowledge management literature pays particular attention to tacit 
knowledge and its role in communities of practice. Tacit knowledge is what an experienced practitioner knows 
internally, but that isn't part of the explicit knowledge held by the community or organization. In a profession as 
complicated and distributed as instructional design, moving hidden tacit knowledge to public, explicit knowledge is 
important to the growth of the profession, and is one of the unstated goals of many communities of practice. 

Participants were asked about tacit knowledge they held — ideas, procedures, strategies or tactics they had 
invented or learned, but that weren't common knowledge in instructional deign. Interestingly, they had little to offer 
at first. It was possible that they were humble or the question was intimidating, but at any rate, participants had little 
to say immediately. 

But elsewhere in the interviews, the designers offered dozens of examples of unique or creative solutions 
and observations about instructional design. Examples of their tacit knowledge included such things as how to deal 
with difficult clients, job aids for applying criteria to projects, or how to gather expert advice to address difficult 
design problems. It just wasn't thought of as tacit knowledge. Instructional designers are problem solvers, and they 
invent solutions as a routine matter in their daily work lives. But they don't necessarily see that this type of 
knowledge as significant or substantial, nor do they think of ways to move it into the public arena. 

At the same time, it was clear that members of in-house communities of practice constantly probed for and 
shared solutions to design problems they faced in projects. It appears that the process of transforming tacit 
knowledge into explicit, shared knowledge is an informal, serendipitous occurrence in the organizations we’ve 
included so far. These are healthy organizations that promote interaction among designers, and this naturally leads 
them to share new ideas. But one could speculate that in unhealthy or highly competitive organizations, such 
knowledge would not be shared as easily or informally. Tacit knowledge could be seen as strategically important, or 
offering a competitive advantage. 

Repositories of knowledge: We think of communities of practice as groups of people. This is most certainly 
what most instructional designers mentioned when asked about their own communities. However, one 
person considered online resources to be an important, if unidirectional, part of the community of practice. Can 
shared resources and forum archives act as communities of practice? They don't fit formal definitions easily, but 
they certainly can be thought of as repositories of thought from communities of practice. There is really very little 
difference in searching a repository of information and lurking in a chat session. If one is primarily an observer, that 
action can still satisfy the requirement of participation in the community. 

Virtual Communities of Practice 

It was clear that participants relied heavily on online repositories of information, and to a lesser degree, 
participants mentioned that they participated in active online communities of practice. These included listservs and 
email lists for the most part — asynchronous communication environments. Comments from the participants 
suggested that they viewed virtual communication and interpersonal communication as functionally equivalent 
(Allen;31). 

The people interviewed are experienced users of technology, and they view virtual technologies as another 
mode of expression, with concomitant strengths and weaknesses. One can speculate that instructional designers, as a 
group, are probably more experienced and sophisticated with using online communication technologies than most 
other professional groups. Given that many instructional designers are creating online learning resources, it isn’t 
surprising that they see virtual communication as a natural extension of their face-to-face communities of practice. 

Not that there aren’t limitations; indeed, one participant struggled with the uncertainty that accompanies 
electronic communication. Uncertainty and ambiguity plague virtual communities of practice. For example, if 
someone asks a question or makes a comment in a virtual community of practice, and there is a limited response 
from other members of the community, the silence can be interpreted by the sender in a number of ways. 

Similarly, it is misleading to speak in gross terms of the characteristics of "online communities." They 
differ from each other as much as they differ from other kinds of communities. They differ in tone, intensity, 
usefulness, number of subscribers and content among other things. In these interviews, participants 
described general networks, professional resource repositories and forums, intranet discussion groups in 
organizations, student societies and email exchange groups. It is important to keep in mind that when we 
discuss online communities, we are really talking about a wide array of entities that operate on several levels and 
within a bewildering number of contexts. 
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Purpose: Virtual communities of practice, just like other communities of practice, don't just happen; they 
form when individuals share a purpose that has importance either personally or professionally. The clearer and more 
important the purpose, the stronger the adhesion of individuals to the group. In education, and perhaps in virtual 
communities of practice, we sometimes make the mistake of thinking that individuals will be drawn to a community 
merely because it is there and people have interests in common. Of greater importance seems to be the issue of 
whether the community will contribute something tangible to the participants. This "social capital" may be common 
to the members of the group, but it is more likely that a strong community will speak to the individual needs of 
participants, needs that hover around a central purpose, but that take on a variety of meanings for individuals. At any 
rate, it is vitally important that communities of practice can answer the question, "What will I get out of it?" to be 
successful (Allen:64). 

Managing online communities: Online communities can be managed differently than most face to face 
communities, mainly because of the control and anonymity provided by technology. For example, one can filter 
messages from an online community and look at them when there is time, or one can route messages to a dedicated 
mailbox. Some online lists permit participants to receive a digest (single message) that contains all of the messages 
on the list that day. That can be very useful for managing time, but presumably reduces the immediacy and 
spontaneity of the interactions with the groups. At any rate, designers reported managing their interactions online in 
skilled and deliberate ways (Allen;38). 

Trinity— people, context, content: The context of a virtual community of practice and the people involved in 
the community are most important to its success according to one participant. But this position was qualified by 
others, who identified content as the central feature that drew them to communities. The impression this left was that 
for virtual communities, all three things are vital. Content is of course important — it provides the substance around 
which discussions form. But in virtual communities, it appears that participants are looking for more. They are 
looking for a context that is convenient to use, and they are looking to expand the number of qualified people they 
can engage, or access to leaders in the field who are otherwise inaccessible. This trinity forms much of the social 
capital of virtual communities, the glue that holds them together (Allen:66). 

Receiving a great deal of advice is typically desirable, and when several people give similar advice, one 
interprets the advice as more reliable. By definition, it is, but caution was expressed about the seeming reliability of 
converging advice in a virtual environment. Contrary advisors may be silent. In fact, silence may cloak several 
different interpretations of the item under discussion (Allen: 57). Therefore, it is reasonable to think of converging 
ideas as less representative of the population from which they are drawn. 

Anonymity and social distance: An interesting issue is the role of social distance in building credibility. A 
common joke is that an expert is someone from more than 25 miles away with PowerPoint. There are old saws like 
"absence makes the heart grow fonder" and "familiarity breeds contempt." The participants made some interesting 
comments that suggested something similar happens within a virtual communication system. Is the development of a 
strong community of practice necessarily dependent on participants becoming more intimate and reducing the 
amount of social distance? Maybe there are advantages to promoting a certain amount of distance. Perhaps social 
distance contributes to the attraction of some members to a community of practice, and thereby strengthens the glue 
of the community (increases the collective social capital of the group). One individual mentioned very clearly that 
anonymity is an important part how he participates in virtual communities (Allen: 69). It is important to this person 
that he is able to dip into and out of a community, use it for what he needs, and then leave. In professional 
communities of practice, he isn't looking for personal or intimate associations. 

At first glance this seems counter-intuitive; we think of communities as places where people make personal 
investments. But that isn't necessarily the case for professional communities of practice. In these settings, the 
agendas may be more utilitarian, at least for some participants. In all cases, it is clear that personal agendas will 
shape the nature of communication in a virtual community of practice in much the same way as personal agendas 
influence temporal communities. 

Social Change and Agency 

There was a clear message that there was a need for the types of social change that ID can provide, and 
evidence of the influence of ID generally on institutional policies and issues. There were also a number of 
frustrations and worries that were articulated about the lack of influence instructional designers felt in their 
organizations. 
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There are some really huge issues that are moving forward in distance education, 
especially technology- enhanced learning issues. If the institutions-the academies-do not 
look at these issues very seriously, very soon, they're going to find themselves in policy 
nether land, where nothing works. (Barbara:62) 

Barbara’s comment warns of a problem identified by Kowch (2003), where the traditional policy-making 
structure in most institutions of higher education are not positioned to address emerging pan-institutional issues 
effectively; that addressing pressing, multi-dimensional socio-political issues requires changes that fundamentally 
challenge the policy-making structures of institutions. In this section of the paper we will consider the kinds of 
social change that instructional designers identified, the kinds of contributions they make to them, and some of the 
unresolved issues that organizations face. 

Social Change: Types and Examples 

Instructional designers described different types of social change that they influenced, and for convenience 
we group them into institutional issues and professional issues. By institutional issues, we mean those things that 
engage the organizations or structures of society; by professional issues we mean those things that engage 
individuals and groups, independent of their organizations. 

Institutional issues: Designers mentioned projects that had wide influence, such as changing social 
attitudes about stereotyped groups (Deborah;54). They also mentioned institutional growth in the prominence 
of instructional design positions (Allen;75), different levels of responsibility to clients, learners and humanity 
(Deborah;53), and a growing appreciation for instructional design by management (Eric:46). 

Professional issues: Participants stated that they had seen evidence of transformed practice among 
educators as a result of their involvement in instructional design, particularly on technology-enhanced learning 
projects. These were often characterized as a shift from content-centered design to learner-centered design 
(Barbara:79). But changes were noted in how instructional designers encouraged organizing and structuring learning 
by modeling good practice (Deborah;58) and focusing attention on important learning outcomes (Eric:33). 

A consistent message was that as instructional designers, people felt they were able to influence larger 
groups of people than they could in other educational roles (Deborah:20). But there was a measure of caution, 
expressed as a concern that in order to have a sustained influence over an extended period, one would need to be 
working on content areas that they felt were important. Perhaps, this suggests that the supposed content-free nature 
of ID doesn't permit designers to connect to substance they value. Is the personal value of what we contribute 
significantly bound with the content areas in which we design? Do designers hunger to work on content they value? 

One of the most provocative statements spoke to the irregularity of personal impact. It is likely that 
instructional designers feel as though their contributions are uneven. They make an important difference sometimes, 
but not necessarily often enough to feel a high degree of personal satisfaction (Allen:72). When we consider the 
effect of ID on social change, and look for large influences, we may overlook the fact that small contributions may 
have a large effect in the long run (Deborah;53) 

This may be especially true of ethical stances and higher values, and how holding to them can have 
profound effects. Perhaps humility about our influence is reasonable and sufficient, even admirable. Instructional 
design may not be so important on a grand scale, but the contributions made can have wide and profound influence 
in the long run. Eor example, if we insist on gender-neutral language, we may in the long run, contribute to a new 
understanding of equality. 

Designers know that they have a great deal to contribute, and that they make a big difference in the 
quality of instruction they influence. But they work in a shadow profession, one that is not fully understood or 
appreciated by those in management. In order to be effective in promoting social change, instructional design needs 
to clarify the kinds of contributions it can make, and make other educators aware of those contributions. It isn't 
enough to work quietly and effectively in the shadows, and hope that the profession is understood and appreciated. 

Issues Not Mentioned 

Schwier (2001) posited that learning communities had a future orientation, an idea complemented by 
Wenger’s trajectories of participation in communities of practice (Wenger, 1998). In the interviews with 
instructional designers to date, little or no discussion of the future direction of the profession or its communities of 
practice emerged. There is immediacy, a descriptive rather than interpretive tone to discussions about the practice of 
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ID. Where is it going? What is its direction? Where is it leading? These are items about which the discussants were 
less vocal. 

The interesting question, though, is why were participants reticent on this issue? It is entirely possible that 
the questions and directions of the interviews did not encourage this kind of observation. Maybe the participants 
were thinking so carefully about describing the nature of their work that they didn't consider whether the 
communities of practice had a future orientation. On the other hand, it is possible that this is characteristic of 
instructional design. Designers work so closely and intensely in the immediate, that they fail to consider the 
direction they are following to the future. 

When asked directly about the significance of instructional design, individuals spoke about their influence 
on immediate projects, or reflected on specific projects they influenced in important ways. Nobody spoke of how 
instructional design might be influencing the future of education or training, at least, not specifically. 

Is this a story of power and influence in the educational hierarchy? Was Heinich (1998) right when he charged that 
in our profession we have aligned ourselves with labor when we should have aligned ourselves with management? 
More importantly, how is a profession sustained? What role does a sense of larger purpose play in nourishing a 
profession and its participants? Although there are certainly exceptions. Medical doctors view themselves as healers, 
lawyers see themselves as protectors of justice, teachers see themselves as improving the future through children. 
What is the simple vision of instructional designers? What is their larger purpose? What is their grand agenda? 

Summary 

This study unveiled a number of intriguing issues and challenges about instructional designers and their 
communities of practice, and how instructional designers perceive their participation in sweeping changes underway 
in education. We see a profession that knows itself, but is struggling for identity and acceptance in the larger 
educational community. We find that instructional designers invent and participate in communities of practice in 
ways that challenge our understanding of models of communities. We learned that there isn’t a single or global 
community of ID practice, but rather, hundreds of small, local, effervescent and convenient communities of practice. 
We see instructional designers struggling to identify their own tacit knowledge, and without systematic avenues for 
sharing their tacit knowledge with other designers. We find that instructional designers recognize that they have a 
role to play in the changes currently underway in education, but less understanding of how to express that role 
forcefully and demonstrate leadership. We see that the focus of designers is institutional more than societal, but that 
they exhibit high standards of performance and care for the appropriate integration of technology into learning 
environments. 

We emphasize that this is a preliminary study. We are encouraged by the high degree of professionalism, 
and the thoughtful reflection designers shared with us. If the participants in this study are representative, 
instructional design is a profession populated by highly competent, intelligent and dedicated individuals who are 
looking for a way to influence important changes in education and society. 
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Abstract 

The presentations within this session originated from a graduate course that explored research and theory 
related to techniques for scaffolding higher order thinking. The first paper describes the course participants’ 
experiential accounts of using webpublishing technology for supporting their reflection. The other papers present 
research-based reviews on teaching and learning of higher order thinking in technology-enhanced environments, 
including: challenges of changing instructor roles, specific techniques for scaffolding student performance, and 
challenges and options for assessment. 



Paper 1: Supporting Reflection Within and Beyond the Higher Education Classroom 

(Priya Sharma) 

The purpose of this study was to explore and describe participants’ experiences of various reflective 
activities and tools within the context of a graduate course. Participants were interviewed and participant data were 
scrutinized to identify initial thoughts and experiences with the activities within the course. 

Background 

The context of this study was a research seminar for graduate (doctoral and masters) students. Within the 
course, students were asked to develop a research framework or a literature review paper within their specific area of 
interest. Developing such a product is characterized by the need for students to engage in prolonged sense-making 
within an unstructured and extensive domain of literature. To delineate the scope of information, students needed to 
identify, gather, and organize resources. Simultaneously students would need to examine the resources to identify 
utility and applicability to their areas of research. The identified tasks call for high levels of reflection and self- 
organization, which sometimes needs to be more overtly triggered and supported. Brockbank and McGill (1998) 
suggest that encouraging reflective and critical thinking in higher education requires sustained effort from all 
participants — including the facilitator. In many cases, this means a significant structural and pedagogical 
reorientation of the learning environment. The traditional classroom setting can support significant reflective 
dialogue but in many cases, the dialogue stops at or tends to progress very slowly beyond certain spatial and 
temporal boundaries. To support this extra level of dialogue and reflection outside the classroom, a type of 
technology known as personal webpublishing was implemented. Webpublishing systems allow individuals to 
chronologically record and archive postings on the Web. Many Webpublishing systems are available for use and 
they are mostly characterized by their ability to allow an individual to edit and post from any networked computer 
with access to a browser and the Internet, thus radically simplifying web authoring. We used a specific type of 
Webpublishing system known as a Weblog. Individual Weblog posts can be of various lengths and they provide a 
mechanism for timely logging of thoughts and resources, thereby allowing an individual to build a repository of 
information. 

Procedures and Setting 

The course was structured around face-to-face meetings that were scheduled every week for the first two 
months of the semester. The face-to-face meetings were wholly discussion based and no lectures or presentations 
took place. Participants engaged in discussion that was initially based on a set of common reading materials but 
which later became more individualized. In addition, each person in the group posted reflections, resources, and 
other types of information on their individual Webpublishing spaces. 
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The group collaboratively established a consensus on schedules and a set number of postings per person. 

An initial set of categories was also used to guide the logging process. Thus, content could be categorized as a 
resource, a commentary, a question, a reflection, an action, or as miscellaneous. Interviews were conducted at the 
end of the semester and interviews averaged about an hour in length. Interview questions focused on exploring 
participants’ perceptions of using the Weblogs and its value in supporting their reflections. Tentative initial findings, 
from an evaluation and implementation perspective, are presented in the next section. 

Need for security 

Because weblogs are public and can be viewed by anyone with access to the Internet, there was an initial 
focus on refining the posts to avoid grammatical and structural errors. Participants admitted that it was an effort to 
overcome the natural inclination to refine and rewrite their posts so that it was suitable for public viewing. Two 
participants admitted that they were “afraid” to expose their thoughts and be considered as “stupid” by external 
readers. Since participants remained uncomfortable with the idea of posting their thoughts in public the logs were 
made private. Soon after the weblogs were made private, the number of postings increased significantly for at least 
one participant who had expressed initial discomfort. 

Importance of face-to-face support 

Although students professed to high levels of interest in using the Weblog system, the face-to-face 
interactions were important to bolster student motivation to use the Weblogs and also to maintain a more personal 
interaction and discussion element. Often, class interactions became fodder for further writing on the Weblogs and 
conversely, the Weblog content became a topic for discussion within class. This combination of two types of 
expression (verbal and written) in thinking and discussion was admittedly appreciated by the students. In addition, 
the in-class sessions seemed to function as a mechanism for clarifying and extending feelings about Weblog usage — 
for example, during the first few weeks of the course, significant time was spent talking about the time and effort 
involved in using the Weblog and participants perceived that those discussions contributed significantly to their 
ability to understand the purpose of the Weblogs and also to sustain reflection. 

Problems of sustained interaction 

The Weblogs displayed a considerable disparity in number and depth of posts made by each person. Some 
participants spent significant amounts of time to produce very in-depth and reflective postings and they also 
consistently posted a larger number of posts. Other participants did not post as often or in as much detail. Supporting 
consistent logging emerged as one of the biggest issues from this pilot. Participants identified two major barriers to 
using the Weblog — time and established modes of reflection. The perceived paucity of time emerged as a significant 
factor in diminishing Weblog activity — one of the reasons, was the natural progression of an academic semester. 
Participants admitted that as mid-semester approached and exams and other assignments became pressing, they 
found it difficult to sustain posting. Even when other assignments were not at the forefront, the amount of time it 
took to read and then document thoughts on the Weblog seemed to be a significant problem. This problem was 
intensified because many of the participants had already established other means of reflection, such as personal 
diaries and or reference databases and the Weblog posting became an additional and unhelpful step for reflection. 
One participant, realizing this early on, stopped posting quite early in the semester, while others persevered. 

Progression in type of logging 

Some participants indicated that they noticed a significant progression in their logging, where logs moved 
from being more resource-based to include more comments and reflections. Initially, many of the logs contained 
lists of resources in terms of bibliographies, excerpts from articles, citations to web links, etc.. As time progressed, 
the proportion of the logs changed from just listing resources to commenting on the quality and applicability of the 
resources to individual projects. In addition, there tended to be a higher proportion of reflective postings and 
participants felt that they were revealing their positions on a topic more openly and with less fear of being “hurt.” 

Lack of interaction 

Participants cited the lack of interaction between individual logs as one of the most disappointing 
occurrences within the course. Although the face-to-face sessions were marked by intensive discussion, debate, and 
interaction, there was no similar occurrence on the Weblogs. Although one participant felt that the interaction was 
sufficient, the other four participants felt that more interaction would have added significant value to their own 
thinking and to their final product. 
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Tentative suggestions for next steps 

Based on the Weblogs postings, participant interviews, and in-class observations and conversations, from a 
design perspective, there are some indications that Weblogs can support reflective thinking and self organized 
learning. The initial analysis indicates the definite existence of questions regarding micro-levels of intervention — 
specifically, how to guide the use and practice of personal Webpublishing systems, how to sustain consistent 
externalization of thoughts on the Weblogs and how to combine these reflections with existing modes and personal 
practices established by individuals, and also how to encourage and sustain interaction between participants on the 
Weblogs. These emerged as the most significant issues cited by participants as having influenced their use of the 
Weblogs. 

Paper 2: Strategies to Promote Reflective Thinking in Problem Based Learning (Husra 

Gursoy) 

The continuous changes in the society create considerably complex problems in various fields and therefore 
it has been recommended that graduates of higher education improve their “thinking skills” that will help them to 
cope with these complex situations. The capacity to engage in reflective thinking is one of those skills which need to 
be improved in higher education settings. Although there are various ways in which reflective thinking can be 
promoted, problem based learning is one approach that provides learners with an appropriate setting that can 
increase their reflecting thinking while exploring authentic and ill structured problems (Albanese & Mitchell, 1993; 
Hmelo & Ferrari, 1997). However, without appropriate scaffolding tools, students have difficulty engaging in high 
level reflective thinking (Hmelo & Lin, 2000). It is important to select appropriate tools to help learners motivating 
in reflection. This paper will summarize best strategies to support reflective thinking in problem based learning 
environments. Various scaffolding strategies and their effectiveness in PBL environments will be discussed and 
suggestions will be provided for future implications. 

Reflective thinking 

The reflective thinking movement began with the studies of Dewey. He defined reflective thinking as “an 
active process” rather than passive, means that one thinks things through oneself by asking questions, finding 
relevant information and eventually by taking responsibility of one’s own intellectual development, instead of 
merely learning from others. Dewey (1933) also delineated reflective thinking from simple “thinking” (such as 
daydreaming, inventions and beliefs) by indicating that it is goal oriented and aims at a conclusion. Schon (1987) 
defined reflection as thinking activity for an extended time about a set of recent experiences by looking for 
commonalties, differences, and interrelations. Similarly, Hatton & Smith (1995) identified some essential issues 
concerning reflection. They expressed that, during reflection, an individual should learn to frame complex or 
ambiguous problems, try out various interpretations, and then modify his actions consequently. Reflective thinking 
provides learners an opportunity to control their learning by assessing what they know, what they need to know, and 
how they bridge that gap during learning situations. Reflective activities also encourage students to analyze their 
performance, conceptualize the actions they used in similar situations, and compare and contrast these actions to 
those of novices and experts (Collins, 1990). 

Problem Based Learning 

In general, PBL offers a unique environment to learners to promote their reflective thinking. During PBL 
session, learners generally confront ill- structured problems and attempt to find meaningful solutions. When learners 
are faced with complex problem solving situation, reflective thinking helps them to become more aware of their 
learning progress, choose appropriate strategies to explore a problem, and identify the ways to build the knowledge 
they need to solve the problem. In order for instructors to facilitate the process in different stages of problem 
solving, scaffolding strategies should be integrated into the learning environment in order to assist students develop 
their ability to reflect on their own learning. From the instructor's point of view, promoting reflection should result 
in having learners think more deeply about a topic, thus develop a more complex and interrelated mental schema. 

In problem based learning environment, learners confront with different stages. Barrows (1994) defined 
four different stages of problem solving process. First, the problem is introduced and presented, and then students 
discuss the problem, generate hypotheses based on their own experiences and past knowledge, identify relevant facts 
and learning issues. Second, students engage in self-directed learning and search the relevant resources through 
libraries. Third, students begin to evaluate the sources they have identified - what was useful and what wasn't- and 
then begin to work on the problem with a better-conceptualized idea. Last, students assess their work by self and 
peer evaluations at the end of the process. 
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In order to better present strategies and scaffolding tools to promote reflective thinking in PBL, we will 
reconstruct these stages with a different perspective. While Barrows stages are iterative and one depends to another, 
the stages defined in this paper will categorize problem based learning activities at group and individual level, by 
specifically focusing on reflection. That is, we will examine how and which forms of reflection occur in group and 
individual problem solving process. We will evaluate the role of reflection separately in each of these three 
situations. These stages can be defined as follows: (1) Reflection on Group Problem Solving, (2) Reflection on 
Individual Problem Solving, and (3) Post Problem Reflection. 

Reflection on group problem solving 

People construct meaningful and sound definitions of social situations and validate their conclusions by 
comparing themselves to others (Fazio, 1979). Groups seek and process relevant information both cognitively and 
socially (Forsyth, 1999). In PBL, groups not only use one another as informational resources but also work towards 
a set of shared objectives. Students in tutorial group collect and share the resources, set goals, and solve the 
problems together. Each member should be aware of the group process, how the group behaves and acts toward the 
solution of the problem. A critical aspect of problem solving in ill-structured situations is that people hold multiple, 
and sometimes conflicting, perspectives of the nature of the problem, procedures for solving it, and appropriate 
solutions (Kitchener, 1983). For this reason, problem solving methods of instruction in ill structured situations 
typically use learning groups. These groups provide a means for students to share and examine others’ 
interpretations and perspectives as they work through the problem (Will, 1997). Two strategies that can help for 
group-problem solving are reflective Journals and group reflection and discussion. 

Reflective journal is defined as a documentation where each individual write about their learning process in 
particular situation or an event and make a personal sense of these experiences. One way to follow up group 
problem solving process in PBL is to have each group member submit a confidential reflective journal to the 
facilitator that describes (1) what the individual’s contributions to the project and the group processes were (2) what 
each other group member’s contributions were (3) how the group process are going (Nelson, 1999). This interaction 
might take place during the learning activity via dialogues or it may occur later in either formal or informal 
discussions. Reflective dialogue and discussion during PBL session occur between both group members and 
facilitator. By sharing ideas, groups and individuals would have clearer understanding of the problem with new 
ideas and insights to consider. 

Reflection on individual problem solving 

Each individual processes problems differently. Therefore cognitive process of problem solving is difficult 
to understand or further define. Even though group members and instructor make important contributions towards 
the solutions of the problem, each group member needs to construct his/her own understanding of the problem as 
well. According to Jonassen (1997), problem solving has two important characteristics. Eirst, learners construct 
mental representation of the problems, which is known as a problem space. Second, problem solving requires active 
manipulation of the problem space by building models, generating hypothesis, testing solutions and gathering 
information. Problem solving also requires knowing what to and how to monitor one’s performance. (Lester, 1994). 
A major significance in solving problems occurs when students step back and reflect on how they actually solved 
the problem and how the particular set of strategies might be improved. Creating suitable representations of the 
learning process in a rich context is a key point to promote reflection of the metacognitive processes. This could 
help students to increase: (1) awareness of the learning process; (2) their metacognitive skills; and (3) their capacity 
to transferring skills to new learning situations (Lohman, 2000). Through metacognitive skills, the learner forms a 
mental representation of the problem, selects plans and strategies for reaching the solution and identifies the 
obstacles that slow down progress (Davidson and Sternberg, 1995). In order to practice problem solving process at 
individual level, several reflective activities are suggested including concept mapping, using reflective diaries, 
modeling, and prompting. 



Post problem reflection 

Post problem reflection is necessary to both review the knowledge gained through the problem and make 
the reasoning explicit through the problem. Post problem reflection occurs when the problem is understood and 
solved at both group and individual level. Each group member reflects on their learning experience at individual 
level, and group’s member’s contributions at group level. Here, post problem reflection is categorized as self 
evaluation and peer evaluation. An important element of the problem-based learning is to help students identify gaps 
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in their own knowledge. “When the problem is finally solved, learners can learn much by reflecting back on what 
was learned about both the problem domain and the problem solving process (p.67)” (Reigeluth, 1999a). The 
following guidance would be helpful for students to structure their reflection: 1) Have them compare the problem 
solving process to their past learning experiences. 2) Have them identify the obstacles or difficulties they 
encountered in the problem solving process. 3) Develop criteria to assess the students’ performance on the task and 
their problem solving skills. Have students reflect their own learning based on these criteria. 4) Have them reflect 
their group learning process. Peer evaluations are considered as a significant part of the evaluation process. After the 
problem solving process finished, instructor should request a confidential reflection from each group member to be 
able to identify problems that the group encountered and provide individual feedback during the process. These 
individual reports give chance students to reflect on the problem and the process in which they were engaged. 



Conclusion 

This paper focused on strategies to develop and promote reflective thinking skills, particularly in problem 
based learning environments. Several strategies to enhance reflective thinking have been suggested based on 
common sense in literature. These activities have been categorized into two phases as individual and group problem 
solving process. Reflective activities on both group and individual problem solving have been summarized and 
examples of related literature have been provided. At the end, reflective activities for the post problem reflection 
have been introduced, at both individual and group level. 



Paper 3: Designing a Classroom as a Technology-Enhanced Learning Environment that 
Prompts Student’s Reflective Thinking in K-12 (KyoungNa Kim) 

The explosive growth and development of technology requires new knowledge and learning skills (Lin, 
Hmelo, Kinzer, & Secules, 1999). Educational researchers have been publishing voluminous amount of new works 
associated with learning theories and extensive educational issues regarding technology integration into learning 
environments. Parallel with these trends is the role of instructional designers working with design and development 
of instructional technology. Their goals are to design and develop educational technology systematically and to 
integrate existing technology to produce more effective learning. One of the examples that are associated with this 
view is revealed in a higher-order educational goal. For example, "Students need to learn. ..to be able to state why a 
question or argument is significant and for whom; what the difference is between developing and justifying a 
position and merely asserting one; and how to develop and provide warrants for their own interpretations and 
judgments (Association of American Colleges, 1991, p.l4).” 

In other words, many educators and instructional designers currently espouse the view that a major goal of 
formal education should be to help develop the student’s ability to think reflectively in order to decide which 
sources are useful and reliable to make sense of information. These skills are viewed as vital, not only to help 
students solve problems, but also help to educate them to become life-long learners who are able to flexibly adapt 
learning to new situations (e.g., Bruner, 1993). Therefore, it is implied that a new paradigm shift is to be expected in 
designing a learning environment for K-12 that includes technology and teachers to support learners’ reflective 
thinking. I support the idea that “it is the classroom that the opportunity to foster thought either arises or may be 
created’’ (Hullfish & Smith, 1961) and that formal education takes place in school. As a result, I believe it is proper 
to turn to the classroom environment as one of the primary learning environments when we consider the question for 
fostering reflective thinking/ activity on the part of students. There are research findings that technology can provide 
scaffolding for reflection and that without appropriate support students have difficulty engaging in high-level 
reflective thinking (Lin, et ak, 1999). It is important to look at the design issues and elements that should be 
considered in supportive learning environment design, which includes the design of different technologies and 
strategies. 

Reflective Thinking and Its Scaffolding; An Essential Ingredient in Learning 

It has generally been assumed that learning how to think opens the door wide to enhancing future student 
learning (Hullfish & Smith, 1969). Especially, the explosive growth of technology that includes World Wide Web 
requiring students to learn not only how to use resources to find relevant information but also how to make sense of 
information makes thinking especially important and decide which sources are useful and reliable (Lin et al, 1999). 
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Therefore, in this new learning paradigm something other than “just thinking” is involved. In other words, it requires 
learners to be able to think reflectively and critically to manage this learning process. 

Many educational psychologists and philosophers have proposed various terms and definitions associated 
with higher-level of thinking ability such as “higher-order thinking”, “complex thinking”(Jonassen, 1996), “critical 
thinking”, and “reflective thinking” (Dewey, 1933; Hullfish &Smith, 1961; King & Kitchener, 1994). These terms 
are commonly and confusingly used. In this paper, I will use the term “reflective thinking” as defined by Dewey 
(1933): “Active, persistent, and careful consideration of any belief or supposed form of knowledge in the light of the 
grounds that support it and the further conclusion to which it tends.” Therefore, reflective thinking is viewed as a 
thinking process reflecting specifically to the processes about what has happened. This process leads students to 
assess what they know, what they need to know and how would be the situations and is to make judgments to 
construct learner’s own knowledge. In terms of K-12, it was discovered that young students perceived the learning 
environment as one of the most important factors that prompt and support their reflective thinking (Koszalka, Song, 
& Grabowski, 2002). However, there has been little research into learning environments associated with K-12 
education that support and scaffold learners’ reflective thinking. Therefore, it is invaluable that this study 
investigates what is taking place associated with this endeavor and finds what should be needed for designing 
supportive learning environment for students’ reflective thinking. 

Technology Scaffolding for Supporting Students’ Reflective Thinking 

Technology can play a powerful role in supporting student reflection (Lin, et al. , 1999). From the review 
of the literature, Lin, Hmelo, Kinzer & Secules classifed four types of technology design features that provide 
scaffolds for reflective thinking: (1) process displays for displaying problem-solving and thinking process; (2) 
process prompts for prompting students’ attention to specific aspects of processes while learning is in action, (3) 
process models for modeling of experts’ usually tacit thinking processes so students can compare and contrast with 
their own process in action, and (4) reflective social discourse for creating community-based discourse to provide 
multiple perspectives and feedback that can be used for reflection 

The Classroom as Reflective Community 

With regards to K-12 learning environments, it is the classroom where learners, teachers, and technology 
are in one place. When they are incorporated into an optimal learning environment for supporting learners’ reflective 
thinking, “partnership” among them are required (Lin and Hsi, 2000). Here, in this section, I would discuss 
designing a “classroom” that provide “optimal learning” for students’ reflective thinking. In terms of the classroom 
as reflective continuity, as early as in 1960’s, Hullfish and Smith emphasized the classroom environment as a place 
where the opportunity to foster thought either arises or may be created (Hullfish & Smith, 1961). To illustrate the 
classroom as reflective continuity, they asserted that when we consider the question of fostering reflective activity 
on the part of students, it is in the classroom that the opportunity is provided to foster thought. In addition, they 
stated it is proper that whatever is done in the classroom should occur in relationship to a pervasive reflective 
atmosphere. Coupled with this earlier statement, there are also recent researchers who addressed “classroom” as on 
of the important learning environment. For instance, Collins & Bielaczyc (1999) proposed a theory “learning 
communities in classroom” and it offers some primary methods regarding the culture of the learning environment in 
classroom. Also, they raised a number of issues about the design of learning environments related teaching and 
learning culture of classroom. With regards to an optimal learning environment for supporting learners’ reflective 
thinking, “meaningful learning” is to be taken into account for design consideration. It puts strong emphases on 
learners’ reflective thinking for it to happen as well as requires learners’ interacting with learning environment 
(Jonassen, 2000). 

Strategies and Scaffolding for Promoting Reflection in the K-12 Classroom 

When it comes to fostering reflective thinking in K-12 education, such other potentially important factors 
as teacher’s scaffolding strategy in class (i.e: question prompt, instructional strategies, and so on), reflective activity 
(i.e: making observations, asking question, and comparing their understandings with those posed by others, etc.) 
and collaboration with peers and adults. (i.e: discussion, tutoring, and peer evaluation, etc.) must be considered 
(Douillard, 2002; Koszalka et al, 2002; Wood, Bruner, and Ross, 1976). Primary elements research indicates are 
teachers, tasks, reflective thinking tools, and instructional environments (Koszalka et al, 2002). First, the role of the 
teacher in scaffolding students’ learning is crucial to the development of reflective thinking not only while the 
students are engaged in the task but also in the creation of a learning environment that encourages young children to 
actively play and/or explore the objects and ideas that they encounter (Yelland, 1999). Teachers can prompt 
students’ reflective thinking by asking reflective questions as well as explaining reflective concepts to students 
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(Moon, 1999; Virtannen et al, 1999). In addition, researches suggest there are some other strategies that can 
encourage reflective thinking: providing students with opportunity to have discussion(Brookfield, 1986), inquiry- 
based teaching ( Kroll, 1992), and peer tutoring experience (Wood, Bruner, and Ross, 1974), which require student 
to work together with their peer students and teachers to facilitate students’ learning with scaffoldings. 

Second, the task and learning activities students are engaged in in class, when students are provided with 
authentic tasks involving ill-structured learning activities, challenging work and complex problem solving , those 
tasks encourage reflective thinking during learning (Hopson, Simms, & Knezek, 2001; Stepien & Pyke, 1997). 
Dewey (1933) and King and Kitchener (1994) propose that individuals engage in reflective thinking when they 
encounter problems with uncertain answers, when no authority figure has an answer, when they believe no one 
answer is correct, and when the solution cannot be derived by formal logic. In addition, the use of reflective thinking 
can be promoted while students are engaged in active learning, inquiry and problem-solving (Yelland, 1999). 
Sternberg (1985) also addressed that the use of higher order thinking skills and the use of metastrategic processes 
that requires students’ reflective thinking are characteristics of effective problem-solving environments. Third, key 
types of tools that scaffold students’ reflective thinking such as reflective journal writing, guiding questions, and 
concept maps are important in fostering students’ reflective thinking (Kinchin & Hay, 2000; in Koszalka et.al, 

2002). There is a shared assumption of educators that reflective writing can promote reflective thinking (King & 
Kitchener, 1994; Ross, 1990). 

To give a picture of what learning environments is taking place in practice that support students’ reflective 
thinking, 1 describe two exemplary learning approaches that “fits” within the reflective thinking teaching-learning 
culture and their elements associated with scaffolding and support: Problem-Based Learning (PBL) and Open 
Learning Environment (OLE). The features that support and scaffold learners’ reflective thinking in each approaches 
and how they include reflective thinking will be explored. PBL is one of “the constructivist pedagogical designs that 
are based on the assumption that learning is a product of both cognitive and social interactions in problem-centered 
environments” (Hmelo & Evensen, 2000; Greeno, Collins, & Resnick, 1996; Savery & Duffy, 1994) as well as a 
new student-centered learning paradigm (e.g., Reigeluth, 1999; Barrows, 1988). According to Koschmann, Kelson, 
Eeltovich, and Barrows(1996). 

OLE has placed emphases on “scaffolding” in more specific ways. To illustrate. Open-ended learning 
environments are comprised of four major components: enabling contexts, resources, tools, and scaffolds (Hannafin, 
Land, and Oliver, 1999). Enabling contexts serve as a catalyst for constructivist learning. Usually presented as a 
problem or project, enabling contexts provide learners with a purpose or goal for subsequent activity. As learners 
pursue learning goals, it is critical that they have access to relevant resources to help them achieve outcomes. 
Resources, which can take the form to documents, human experts, databases, for example, are intended to support 
learning. Tools comprise the third component of open-ended learning environments and equip learners with the 
means to externalize what they know. In an effort to provide learners with support while learning, scaffolds 
comprise the fourth components of open-ended learning environments. When learners are in pursuit of learning 
goals, scaffolds help initiate and sustain meaningful learning by assisting learners as they engage in conceptual, 
metacognitive, procedural and strategic knowledge processes (Hannafin, Land, and Oliver, 1999). 



Discrepancies between ideal and practice: Efforts to embrace some practical issues 

1 have discussed three important factors (technology design features, PBL, and Open- learning 
environment) that support student’s reflective thinking. Just as Lin et al. suggested that they eventually could 
develop programs that incorporate all four technology design features in a single system, 1 have hope that those 
above key features supporting learners’ reflective thinking could create an environment where a combination of 
these features is placed in a single system. However, there has been found discrepancy between our anticipation and 
the practice. Dewey already made a statement as early as in 1916 in his famous volume Democracy and Education: 
“no one doubts, theoretically, the importance of fostering in school good habits of thinking. But apart from the fact 
that the acknowledgement is not so great in practice as in theory, there is not adequate theoretical recognition that all 
which the school can or need do for pupils, so far as their minds are concerned (that is, leaving out certain 
specialized muscular abilities), is to develop their ability to think.” There is another discrepancy, which is associated 
with technology integration that is taking place in school. Eor instance, Jonassen (1995) proposed some suggestions 
with regards to supporting communities of learners with technology. He, however, acknowledged that he doubted 
about its practice. He argued that “the most productive and meaningful used of technology cannot occur in 
traditional educational institutions,” and “until we reform our conceptions of learning, technologies will continue to 
be delivery vehicles and not tools to think with.” 
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Given the fact that formal education goes with the classroom, it can be a ground for designing learning 
environments that support students’ reflective thinking, which includes engagement of teachers, students, and 
technology’s role, even though there are discrepancies between ideal and practice. To create a learning environment 
that can support learners’ reflective thinking in K-12, there should be efforts of a confluence and collaboration of the 
strength from technology design features embedded in learning environments design, teachers, and peers, where 
students are supported and scaffolded by those factors that prompts their reflective thinking in an optimal learning 
environment.In addition, more research is needed to investigate what are the features that prompt students’ reflective 
thinking. Also, further research is needed to examine how both instrumental design principles and technology design 
features can be incorporated to provide a learning environment that supports and scaffold teachers’ instruction as 
well as students’ learning in K-12. 

Paper 4: A Review of Literature on Self-Directed Learning in a Problem-Based Course 

(David R. Morales) 

Self-regulation is the learner’s ability to monitor relevant aspects of their learning as it relates to obtaining 
an established goal or standard of performance. Researchers have related self-regulation to a learner’s dependency 
on the instructor for learning (Keller, 2002; Boggiano, Flink, Shields, Seelbach & Barrett, 1993).), as influencing a 
learner’s behaviors (Beston, Fellows & Culver, 2001; Ley & Young, 1998), motivation (Hofer & Yu, 2002), and 
established goals (Schunk & Ertmer, 1999) to engage in new learning environments (Estee, 1999; Lohman & 
Einkelstein, 2000). The focus of this report is on reviewing literature related to the theory of self-regulation in 
problem-based learning courses. Specifically, this report will describe how instructors in higher education support 
learners in efficiently and effectively self-directing learning. 

Problem-based learning evolved from the scientific model for inquiry (Churchman & Ackoff, 1950). 

Inquiry learning is a process where learners interact with the learning environment to deductively or inductively 
resolve questions and hypotheses through evidence based on data acquisition. The methods used for data collection, 
the amount of time and data required to resolve this disequilibria is dependent on the type of questions and 
hypotheses that are generated, and the goals for accomplishing the outcome. Churchman and Ackoff (1950) describe 
it as an iterative model based on the “science ideal.” The components of this model have the learner constructing a 
module, describing the problem, designing the experiment, testing their assumptions, and drawing conclusions based 
on prior knowledge and experience. 

The manner in which a quandary is developed and described will drive interactions, data acquisition, self- 
regulation, and learning outcomes. In addition, the activity drives the learner’s selection and rejection of data, and 
“relevancy and irrelevancy of hypotheses and conceptual structures” (Churchman and Acker, 1950, p. 198). In 
addition. Prior knowledge influences the perceptions and actions of the learners. Learners continually reflect upon 
information and experiences to make judgments when engaged in complex processes (Dewey, 1910; Ennis, 1987; 
Lipman, 1991; Halpern, 1999; Kuhn, 1999). 

Research related to self-direction describes a positive influence on learning goals, self-evaluation, and self- 
regulation on university students enrolled in an introductory course for computer-skill acquisition when goals and 
objectives are explicit (Schunk & Ertmer, 1999). In addition, perceived over control on a learning task negatively 
influences a learner’s self-regulation strategies (Boggiano et al.,1993). Lohman et ak, (2000) studied the influence of 
group size on self-direction and problem solving in a problem-based learning format. The results showed that 
medium size groups of four have the greatest impact on self-direction. 

In conclusion, the literature reviewed for this report shows that clear and explicit goals, motivation, 
learning team size, perceived instructor control, and experience with instructional methods and concepts will impact 
a learner’s self-direction in research based activities. In addition, learners may still be dependent on the instructors to 
direct their learning. If the goal is to have learners develop knowledge and skills that will help them function in 
careers where information and technology are continually changing then it is essential to have students develop the 
motivation to initiate, apply, and refine research strategies. 

Paper 5: An Instructor’s Role Change In Process: Moving Toward a Transformative 

Model of Instruction (Beth Sockman) 

“Instructor as facilitator” has become an accepted role in education (Carr-Chellman & Dyer, 2000; Cheng, 
2001; Hargreaves, 2001; Learning for the 21st Century: A Report and mile guide for 21st century skills, ; Mok & 
Cheng, 2001; K. L. Peck, 2003). However, the transition from the traditional role to the new role can be 
problematic (Billings & Eitzgerald, 2003; Carr-Chellman & Dyer, 2000; Rogers, 1995; Shulman, 1999). This study, 
chronicles one instructor’ s exploration of transition from a transmittive model of instruction, which parallels the 
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instructor as presenter paradigm, toward the transformative model of instruction, paralleling the instructor as 
facilitator role. Using Brockbank and McGill (1998) as a framework for the study, the instructor records her learning 
through journal writing (Lukinsky, 1990) and peer feedback (Brockbank & McGill, 1998). After each instructional 
strategy, a retrospective analysis highlights conflicting emotions, trials and inner resistance. The use of I designates 
the author. 

During the first semester, of teaching an introduction to technology course for educators, I personified an 
instructor’s contradiction. I believed that transformation occurred when unexamined beliefs and actions were 
questioned, and the role of the instructor was to facilitate this process. But, in practice, I had taught the course using 
a transmitter model, embellished with hands on learning experiences. This contrast led to a dissonance, which I was 
anxious to alleviate. In an effort to align my theory and practice, three tactics were chosen for the process: 1 .) 
Keeping a written journal (Lukinsky, 1990) 2.) Reading related literature (Glaser & Strauss, 1967; Lukinsky, 1990) 
and 3.) Participant and observer checks (Creswell, 1998; Lincoln & Guba, 1985; Patton, 2001; S. M. Peck, 1993). 

Analysis focused on identifying and explicating the tension I felt moving from the transmittive model 
toward the transformational model. The term lesson replaces the traditional use of the word theme since each 
experience taught a lesson to me through which I became cognizant of a new discrepancy. Each lesson is situated 
within an instructional strategy that aligned with a transformative model of instruction. Embedded within each 
lesson there is construct entitled, pedagogical discrepancy which highlights the emotions and inner tension in the 
context of the learning experience. Additional literature was intertwined throughout the themes to help make sense 
of the dynamic learning (Eullan, 1993; Eullan & Stiegelbauer, 1992) and provide the next step toward growth. 

Lesson: It is easier to tell than listen 

Context: Students had created rubrics, some rubrics seemed incomplete to me and needed correction. In 
order to correct students in a timely manor, I wrote comments on the rubrics. I had expected to student to address 
questions and to revise their rubrics. To my surprise, no one had a question. In one instance I told a group about 
their mistakes. Upon returning fifteen minutes later, I found that the group had not revised their rubric. 

Pedagogical discrepancy - Transmitter thinking: I told the students directly about their mistakes, 
and spoke with them in a small group setting. Students should have corrected their mistake. 

Transformative thinking: The students did not dialogue; the students did not question their 
assumptions. 

As a strategy to help students question their assumptions, Brockbank and McGill (1998, pp.l 10-104) 
advocate the use of assessment triangulation by the instructor, student and peers. I was the only one giving feedback 
to the students. Instead, students could have offered feedback to one another. It took another lesson to make me 
aware of how deeply “telling” was entrenched in my teaching style. 

Lesson: Be humble and learn from the students 

Context: I slowly learned to listen. A student led a discussion in an on-line discussion where students 
would begin get their “inside out” (Shulman, 1999). The discussion leader’s first question was: “How is knowledge 
of material acquired (for learners of any age)? Eor instance, by classroom lecture, studying bookwork, memorizing 
facts, applying new concepts, personal experiences, a combination of any of these or different contributors.” 
(Discussion board. Spring 2001) 

A response read: 

I think true knowledge is acquired in many ways for different people. . . .In most cases in high 
school and even in college, the subjects from whom I gamer the most knowledge are those that 
intrigue me, for the others I often have found myself memorizing. It is hard to intrigue everyone's 
interest when teaching, but there are certain ways that most students seem to acquire more 
knowledge. Often memorizing facts, reading the textbook and listening to lectures are the least 
affective ways to get someone to obtain knowledge. (Student response to discussion board. Spring 
2001) 

The thoughtful responses surprised me. Eor the first time, the students spoke about their personal 
experiences. Yet after reading the students response, I felt threatened. 

Pedagogical discrepancy: Transmitter thinking: I am looking at the discussion board and am 
intimidated. The thoughts being generated are highly thoughtful, insightful, and erudite. I think 
this (my intimidation) comes from the need to “give an answer” or at least have an answer in my 

head where I am leading the students How can I help them make clear to themselves, their 

sense of logic - not necessarily bring them to my logic? (Journal entry. Spring 2003) 
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Slowly, I began to listen and use guidelines for facilitating discussion (Brockbank & McGill, 

1998; Brookfield, 1995; Browne & Freeman, 2000) and giving students more choice. 

Lesson: Grading the end product or acknowledging the risk? 

My last challenge dealt with assessment while grading a particular student. He had created a web page that 
was incomplete. Unlike the rest of the class, he had experimented with various features such as using flash buttons 
and frames. He had gone beyond the work of anyone else in the class, but had not finished the web page. 

Pedagogic discrepancy - Transmitter thinking: The student did not finish all of the criteria I had 
established for the course. A grade means upholding a standard. Transformative thinking: The 
student already knew how to make web pages and instead took a more courageous route. He 
completed all other assignments and always questioned. Assessment should reflect the students ’ 
risk taking, not only my pre-prescribed standard. 

Throughout the semester, I had become acutely aware of my own failures in order to grow. Writers on 
innovation in the new economy discuss failure as a goal in experimentation in order to exploit creativity, boundaries, 
skills and eventually create successful innovations (Matson, 1996; Peters, 1997). In science, more time is spent with 
failed experiments than with successful ones. These beliefs and experiences influenced my grading. 

Assessment may be the most important factor influencing what and how students learn (Lundquist, 1999; 
McMahon, 1999). Lunquist (1999) charges that focusing on results inhibits the learning process. He explains that 
learning requires experimentation, hypothesis and conjecture. If instructors are results oriented, students tend not to 
take risks. In fact, there is a disincentive for taking risks because risks are time consuming and may not yield 
results. Again, I had not completely learned from the first lesson in the course. We should have attempted to 
triangulate our final assessments. Negotiation of assessment is recommended for moving away from an 
authoritarian structure for assessment and helps learners engage in critical reflection (Brockbank & McGill, 1998; 
Lundquist, 1999; McMahon, 1999). Risk taking, self monitored learning through self reflection and peer input are 
critical as we prepare teachers to take on the challenges in education(Cheng, 2001; King, 1999; Learning for the 
21st Century: A Report and mile guide for 21st century skills, ; McMahon, 1999; Mok & Cheng, 2001; K. L. Peck, 
2003). 

Discussion of the Lessons 

I sought to explicate a personal struggle of an instructors role change as I moved toward a transformative 
model of instruction. Learning was a slow evolution making small changes in a chaotic fashion in response to my 
interpretation of learning and resistance. Some learning took on the spiral nature (Reigeluth, 1999). Multiple failures 
were necessary to correct and substantiate a need for change. The first two instructional strategies yielded similar 
lessons; Students needed opportunities to speak about personal experiences and beliefs. It was not until I began 
using the on-line discussion board that I began to listen and they began to tell about their experiences. Learning 
about assessment also took on a spiral nature. Although I was cognizant of triangulation for my own reflection, I did 
not apply triangulation to the student’s learning. The on-line discussion board took on a dual nature for the class and 
me. The students had the time to take a stand and synthesize their personal experiences with topics relevant in 
education. This was consistent with Hawkes’ findings (2001) where teachers used a discussion board to engage in 
reflective practice more than in than in face to face interaction. 

In addition, the discussion board provided a scaffolding tool for me (Palinscar, 1986). After the first two 
lessons, I was aware that I needed to listen to the students (Brooks & Brooks, 1993), but conflicting issues inhibited 
this from occurring during class. Since I did not feel compelled to give an immediate response, I was able to look for 
areas that would encourage critical reflection in the students and could practice questioning techniques in writing. A 
substantial aspect of the learning process dealt with emotional and psychological issues(Fullan, 1993; Fullan & 
Stiegelbauer, 1992). As Zaltman and Ducan (Ellsworth), posit some of my personal resistance was due to 
psychological forces such as feeling threatened by student ideas and becoming discouraged by my own mistakes. In 
one journal reflection I wrote: 

Old habits die-hard and new ones seem to have a long gestation period. . . . One student asked - 
why do so many professors rely on lecture/read/write format if we don’t learn best that way? 

Why? Newness threatens our identity- threatens who we think we are. I think being an instructor is 
as much about personal transformation - getting outside of yourself and thinking how learning 
occurs- as it t is about teaching others. This is so much harder said then done. (Journal, Spring 

2003) 

Internal tension arose while reflecting during and after each class. Every time I attempted to implement a 
strategy, I would feel dissonance. Controversy has been called an exciting pedagogical vehicle (Browne & Ereeman, 
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2000). Although it creates a state of discomfort, it can be a healthy uneasiness (Brockbank & McGill, 1998). 

Writing in a journal provided a safe place for me to honestly address the dissonance and wrestle with the tension 
between roles. 

By explicating the process, it is clear that I am still moving toward a transformational model of instruction. 
Perhaps, if I had arrived, it would not be a transformation. As Freire notes (1998), to learn proceeds to teach, and to 
teach is the “fabric of learning” (p. 31). Therefore, I will continue learning while engaging in reflective practice, but 
submitting oneself to the process of questioning (Freire, 1998) takes intellectual courage. By intellectual courage, I 
mean the ability to make the academic-self vulnerable to the emotions, the inward struggle, and a community of 
learners in an effort to improve pedagogical practice. As instructors in higher education, we have the opportunity to 
shed the old paradigm and explicate the process. Perhaps if we examine our own practice, we can better empathized 
and have a deeper understanding of what we are asking others to do when making the role change. 
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Abstract 

In order to approach the National Educational Technology Standards (NETS), teachers must be 
significantly better prepared to use technology effectively in their classrooms. The Tech4u.org website is designed to 
provide online professional development incorporating content modules based on NETS and offering the 
opportunity for teachers to create an original “Teaching with Technology” electronic portfolio. The effectiveness of 
this professional development is examined in this study, which is best described as a work in progress. 

Introduction 

In an effort to establish national standards for teachers using educational technology in the classroom, the 
International Society for Technology in Education (ISTE) noted significant benefits to the use of technology in the 
classroom, enabling students to become: 

• Capable information technology users 

• Information seekers, analyzers, and evaluators 

• Problem solvers and decision makers 

• Creative and effective users of productivity tools 

• Communicators, collaborators, publishers, and producers 

• Informed, responsible, and contributing citizens 

They further emphasize that “to live, learn, and work successfully in an increasingly complex and information-rich 
society, students and teachers must be prepared to use technology effectively” (p. 4, 2002). Unfortunately, in the 
public school setting, large numbers of students complain that their teachers are not IT literate, in that teachers do 
not use technology to its fullest capabilities in class or in assigning work (Regan, 2002). 

The National Educational Technology Standards for Teachers (NETS-T) provide a vision of teachers who 
empower students with the advantages technology can bring, who are equipped with technology resources and skills, 
and who can effectively teach the necessary subject matter content while incorporating technology concepts and 
skills (ISTE, 2002). If teachers are to embody this vision, they must have opportunities for meaningful professional 
development experiences. The Tech4u.org website (http://tech4u.org) is designed to provide online professional 
development incorporating content modules based on the NETS standards and offering the opportunity for teachers 
to create an original “Teaching with Technology” electronic portfolio. 

In an electronic portfolio, teachers can demonstrate their effectiveness in the skills of technology use, as 
well as their innovation and experience with impacting student learning in their classroom with technology. 
Portfolios have been widely accepted as effective student assessment tools since the early 1990s. Later, the 
“assessment only” role of the portfolio began to evolve into a focus on artifacts as evidence of learning (Wilcox & 
Tomei, 1999). In Teacher Portfolios: Literacy Artifacts and Themes, Rogers & Danielson (1996) emphasize the 
benefit of deep understanding through reflective and critical thinking that can come over time as a teacher develops 
a professional portfolio. Teacher portfolios may also be used as an authentic assessment to aid administrators or 
supervisors in evaluating teachers (Glatthorn, 1996). 

Purpose of the Study 

The purpose of this study was to determine the effectiveness of the Tech4u.org online professional 
development for teachers. The website contains eight instructional modules: Eile Management, Effective WWW 
Search Strategies, Multimedia, Teacher Tools (word processing, spreadsheet, and database). Online Communication, 
Internet Productivity Tools, Copyright, and Web Development. Each module offers the participant an opportunity to 
gain awareness of technology applications, explore the applications that interest them most, learn technical and 
pedagogical skills, apply learned skills either personally or professionally, and share and reflect on their experience 
with other teachers. The culminating project is to create a “Teaching with Technology” electronic portfolio as 
evidence of their learning and proficiency. Is the course content effective in terms of an increase in likeliness of a 
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teacher to apply instructional technology tools in the classroom increase as a result of participation in the 
Tech4u.org professional development course? 

Methods and Data Sources 

Two groups of teachers completed the online professional development experience - 17 in the Spring 2003 
semester and 1 1 in the Summer 2003 semester. Both of the course offerings were conducted completely online, with 
no face-to-face interaction. Prior to beginning the first course offering, participants completed a pre-survey based on 
the Technology in Education Competency assessment instrument. This nine-item Likert-type survey was adapted 
from Preparedness of Graduates, a national survey on information technology in teacher education conducted by 
ISTE. Upon completion of the course, the Technology in Education Competency assessment instrument was also 
used as a post-survey, with the addition of seven open-ended questions. Due to the minimal amount of information 
gathered from the pre- and post-surveys in the first course, the researchers deemed it necessary to expand the 
instrument prior to the second course to gain a more accurate reflection of the total content learned. Eleven 5-point 
scale questions were added to the first pre- and post-surveys and administered to students in the second course 
offering. As a result of the decision to improve the instrument, only pre- and post-survey data from the second 
course is reported in this study, however, interview data from both courses is taken into consideration. Pre- and post- 
surveys were completed by eight of the eleven students in the second course offering. The surveys measured how 
likely the teacher was to use particular instructional technology tools in the classroom using a 5-point scale ranging 
from 1 {very unlikely) to 5 {very likely). These surveys can be accessed at 

http://fp.okstate.edu/stansbe/surveys/IT pre.htm (pre-survey) and http://fp.okstate.edu/stansbe/surveys/IT post.htm 
(post-survey). 

Personal interviews with a purposive sample of participants from both courses were also conducted to gain 
further insight into the effectiveness of this professional development experience. These interview transcripts, along 
with responses to the open-ended questions on the post-survey, were coded for emerging themes. In addition, each 
student’s completed “Teaching with Technology” electronic portfolio, which was assessed with a rubric, was 
examined in the document analysis phase. 

Description of Professional Development Course 

The course employed two platforms for delivery of information and interaction. Course content, including 
instructional modules and ePortfolio examples were accessed via the Tecli4u.org website. Student product sharing 
via posted assignments and reflection via threaded discussion databases was accomplished using the Blackboard 
learning management system (http://blackboard.okstate.edu). 

The Tecii4u.org curriculum is presented as 8 learning modules — Pile Management, Effective WWW 
Search Strategies, Multimedia, Teacher Tools (word processing, spreadsheet, and database). Online Communication, 
Internet Productivity Tools, Copyright, and Web Development — and were built around the model of the 
Technology Learning Cycle (Diggs, 2001). Thus, each module included the following sections: 

A. Awareness 

Students viewed sample products in the Tech4u.org example ePortfolio and read teacher stories of 
technology use from the Knowledge Innovation and Technology in Education (KITE) database 
(http://kite.missouri.edu). In reflecting on this part of the cycle, students generally responded to the 
questions “What module tools or activities are you aware of? What first caught you eye? What held 
your attention?” 

B. Exploration and Piltration 

Students read a short description of the module content they were about to learn to assist in making 
filtration decisions. Reflection in the Exploration and Piltration portion of the cycle centered around 
what tools were available to students to undertake the module and what products had been created in the 
past that fit within the particular module. 

C. Learning 

Depending upon the module, this portion of the cycle included different types of learning tools 
including online tutorials housed both locally and at other locations on the Internet, interactive tutorials, 
and text-and-graphics based content which provided an opportunity for students to learn to use the 
technology which was the focus of the module. Poliowing the instructional portions, students reflected 
upon the process answering the following questions: “What technical skills related to the module have 
you learned? What technical skills would you still like to learn? What pedagogical applications would 
you like to try?” 
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D. Personal and Professional Application 

Each module required students to create a product using the technology learned in the module. In some 
cases, these products were curriculum materials that could be used in their own classrooms; in other 
cases, the products were simply personal applications of the technology which were relevant to 
students’ lives at the time they were taking the course. Each time, students reflected upon their 
application and the process of creating the product. 

E. Sharing and Reflection 

The final process in the cycle is for students to reflect upon their learning experience of the module as a 
whole and to share with other class members the product they created in the application step. 

The culminating project in the course required participating teachers to create an electronic portfolio 
including at least one self-selected artifact from each of the modules. Additionally, written reflections on each of the 
artifacts were required. 

Both the first offering (Spring 2003) and the second offering (Summer 2003) of the course were taught by 
the same instructor using the same materials, syllabus, schedule, and activities. At the outset, each of the two groups 
of students tended to participate in very much the same way. They would undertake the required readings or content 
interaction, and post their reflections to the database. The instructor experienced frustration in the lack of student-to- 
student interaction at the outset at both of the courses. Using both asynchronous (discussion board and e-mail) and 
synchronous (instant messaging and telephone) instructor-to-student communication tools, the instructor attempted 
to directly task students with the responsibility and value of student-to-student interaction. This encouragement 
resulted in an increase in student-to-student communication and collaboration as well as a sense of connectedness to 
the instructor. 



Findings 

The research question explored in this study was to determine the effectiveness of the course content based 
on based on whether a teacher is more likely to apply instructional technology tools in the classroom as a result of 
participation in the Tech4u.org professional development course. The self-reported likeliness to use instructional 
technology was expected to increase from the pre- to the post-survey, indicating effectiveness of the course. Results 
of the course mean per question and standard deviation is shown in Table 1. 

Table 1: Course Means and Standard Deviation per Question 



Question Topic 


Course Mean, 
Pre-Survey 


Course Mean, 
Post-Survey 


Standard 

Deviation 


Digital file organization 


3.375 


4.625 


-1.25 


Digital file organization with students 


2.5 


3.5 


-1 


Word processing 


4.375 


4.875 


-0.5 


Word processing with students 


3.75 


4.375 


-0.625 


Spreadsheet/database 


3.875 


4 


-0.125 


Spreadsheet/database with students 


3 


3.375 


-0.375 


WWW search strategies 


4.375 


4.875 


-0.5 


WWW search strategies with students 


3.25 


4.5 


-1.25 


Multimedia presentation 


3.625 


4.625 


-1 


Multimedia presentation with students 


3.375 


4 


-0.625 


Internet tools 


4 


4.75 


-0.75 


Internet tools with students 


3.25 


4.14 


-0.899 


Online communication 


4.5 


4.75 


-0.25 


Online communication with students 


3.625 


3.125 


-0.5 


Web development 


3.5 


4.5 


-1 


Web development with students 


2.625 


3.125 


-0.5 


Project-based learning with technology 


3.625 


4.375 


-0.75 


Evaluating/selecting new technology 


4.125 


4.75 


-0.625 


Professional portfolio 


3.25 


4.5 


-1.25 


Learning portfolio with students 


3.75 


3.75 


0 
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The overall pre-test mean of 3.59 increased to a mean of 4.29 on the post-test, indicating an increase in 
likeliness to use instructional technology as a result of the course. A t-test result of 8.66, when p=.01 and df=19, 
revealed a statistically significant increase between the mean score of the collective pre-survey and the mean score 
of the collective post-survey. 

Of particular interest to the researchers is the breakdown of scores by questions for each of the instructional 
modules. The likeliness of teachers to use technology tools themselves was higher than their likeliness to have 
students in their classroom use the same technology tools. Teachers were least likely to have students engage in 
online communication and web development activities, and they were most likely to have students engage in WWW 
searching and multimedia presentation development. Prior to the course, teachers were least likely to have students 
engage in web development and digital file management, and most likely to have students engage in word 
processing, multimedia presentation, and online communication. 

The data collected from open-ended post-survey questions and personal interviews gave a broader and 
deeper picture of the effectiveness of the course content and the teachers’ engagement in the experience. Participants 
found the course to be “very useful and motivating,” “a great course,” “beneficial,” “useful and inspiring,” 
“enlightening,” and “knowledge strengthening.” Some students were very comfortable with and appreciative of the 
completely online format, while others suggested at least partial face-to-face additions to the course. 

Many participants recognized that, while they learned a lot in this class, they also gained an increased 
awareness of how much they had yet to learn. One student in particular reported that they started the course with “a 
thimbleful [of instructional technology knowledge and skill], and now I have maybe 3/4 of a cup.” This theme of 
“there’s ALOT to learn” did not seem to result in discouragement, but rather motivated all of the students to eagerly 
recommend the course to others. 

One important theme that emerged was the recognition of the value of instructional technology’s role in 
pedagogy rather than just tools to be applied. One participating teacher spoke of an intention to “introduce new and 
different instructional techniques to students, incorporating many different learning styles and methods.” Others 
indicated a new belief that “technology can increase motivation and provide a variety of learning opportunities” and 
“there is a lot more I can do and still make the learning meaningful.” Yet another teacher espoused the newly formed 
opinion that “technology is a tool to be used by all teachers - even in PE!” The intention of using web development 
to showcase student work and to show progress at parent teacher conferences was expressed by one participant. 

During the development phase of the curriculum content modules, the researchers included the Digital File 
Management section rather reluctantly, assuming most teachers would already have a good grasp of this. 
Surprisingly, through interviews and open-ended post-survey questions, the participating teachers greatly valued this 
portion of the curriculum. One teacher expressed that they “learned the value of organizing my computer. I will 
never again have 150 loose files in ‘My Documents’.” Another noted that she was now “developing a system where 
I can gain access to my materials and lesson plans without having to dig through a huge mess.” 

For teachers who tend to experience training on technology tools outside of the context of their particular 
classroom or personal interest, this course was appreciated for its relevance to both personal and professional needs. 
The ‘Teaching with Technology’ electronic portfolios created by the teachers were filled with evidence that 
participating teachers were not only competent in using instructional technology tools, but they had also emerged 
from the course with an overall sense of confidence and understanding in their ability to improve student learning by 
both modeling the use of instructional technology tools and incorporating them into learning activities. 

Conclusions 

Findings indicate a high degree of effectiveness of the content offered in the Tech4u.org professional 
development course. The ‘Teaching with Technology’ electronic portfolios served as excellent tools for showcasing 
accomplishments and possibilities. Further research is suggested to determine what changes these teachers actually 
make over an extended period of time in their teaching and learning practices that can be attributed to this course. 
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Abstract 

Pre- and post-evaluations were used to evaluate student’s perceptions of on-line courses. Inquiry focused 
on three primary areas including a.) interaction with course content and instructors, b.) accessibility of resources in 
an on-line environment, and c.) perceptions of on-line course engagement. Findings indicated that students both 
anticipated and experienced positive interactions with course materials, graphics, visuals, and instructors. 
Additionally, students anticipated and experienced adequate access to needed resources including support 
personnel. Finally, students favorably perceived course design components. Overall, students valued the flexibility 
of using on-line technology and thought the course was well organized and met their expectations. 

Introduction 

Facilitating the learning process in on-line environments is a new and important competency for university 
professors. At the University of Houston (UH), the third largest comprehensive public research university in the 
state of Texas, this is a challenging opportunity. The University of Houston has 33,000 students, of which 49.3 
percent are minorities (Smith, 2003). Ninety percent of the students live off-campus and face many obstacles in 
driving to an urban university located near a busy downtown business district. This scenario provides background 
for UH’s distance education endeavors. The Consumer Science and Merchandising (CSM) program, in the College 
of Technology, was one of the first programs to offer on-line content courses. Knowing that facilitating the learning 
process in on-line environments was going to be a new challenge, four CMS professors decided to conduct pre- and 
post— evaluations of their course designs. 

Conceptual Framework. 

Course design is a major component of any distance education initiative. Thompson (1999) stated that close 
attention to curriculum design and delivery is of critical importance to successful e-learning interventions. Porter 
(1997) related that one of the greatest strengths of distance learning lies in the different formats by which 
information can be presented. Goodwin (2000) added to the importance of course design by stating that instructor 
and curriculum quality, whether training in a traditional classroom, two-way video conferencing, or via a web-based 
environment, ultimately determines the effectiveness of a training or education intervention. Pallof and Pratt (1999) 
shared that the instructor has four roles in the on-line class: technical, organizational, social and intellectual. All four 
roles are intertwined with course design, as all roles focus on ensuring that learning occurs. Hall (2000) contended 
that a good curriculum can potentially make up for poor quality media, but it doesn’t work the other way around. 

Given the importance of course design in on-line environments, students’ expectations of and experiences 
with course design is critical to ensure effective learning. Realizing that evaluation can have various foci, such as 
testing of knowledge or skills (Banta, 2002), professors need to ensure that their evaluation efforts also provide 
feedback about their own performance in designing the on-line courses. Students should be given the opportunity to 
reflect on the learning process and provide feedback about the course design. Wade (1999) described student 
evaluations of instruction as “one of our most consistent and, we have traditionally thought most strong indicators of 
course quality” (p. 95). 

Evaluation of student expectations is crucial as it gives professors opportunities to adapt their course design 
to better facilitate learning. Kaynama and Keesling (2000) shared that in developing their web-based Internet 
marketing course, they used a technique called “assessment quality circle” to gather information from students on 
their reactions to the learning experiences. The evaluation technique revealed the need for additional class time to 
work within groups and the need for experts to provide, in advance, an outline of their presentations. This was 
important for design revisions. Kaynama and Keesling (2000) concluded that this integrative evaluation program 
facilitated continuous improvement in the instructional delivery and learning process. Bennion (2002) in his article 
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on evaluation stated that a biology professor was able to bridge classroom experiences and problem solving based 
experiences after receiving feedback in midterm course evaluations that the two types of activities seemed 
disjointed. Bennion (2002) continued that course design is an ongoing exploration, revealing new ways to help 
students achieve faculty expectations. Indeed, creative forms of early evaluations need to be adopted by professors 
of on-line education. On-line courses, because of the lack of face-to-face contact between instructor and student, 
require special efforts in order to obtain information regarding learner expectations and satisfaction. Neal and 
Ingram (1999) suggested that what students have learned and their level of satisfaction with distance learning 
courses remain largely unanswered until the traditional end-of-course evaluation forms are completed and reviewed. 
Special attention must be given to obtaining student feedback before and during the course implementation. 

Summative evaluation, while it should not be the sole point of feedback, it is also critical. While summative 
evaluation cannot impact learning during the current semester (Robles & Braathen, 2002), it can provide suggestions 
on how to redesign the course to best meet the content and process needs of future students. Lee, Owens, and 
Benson (2002) highlighted the evaluation strategies of a web-based MBA program jointly developed by a Fortune 
500 consulting organization and the business school of a major research university. In particular, weekly survey data 
collected from students during the course were content analyzed at the end of the term for overall trends. This 
summative analysis led to a program structure change from four courses per 15- week semester to two courses per 8- 
week mini-term. In evaluating courses from an on-line Master of Education program at the University of Illinois, the 
Instructor and Course Evaluation System (ICES) was used to evaluate the overall quality of the courses. The ICES, 
however, was just one instrument in a broad program evaluation initiative (Levin, Levin, Buell, & Waddoups, 

2002). 

Evaluation is an important element in determining course design quality. While on-line course design is a 
new competency for most professors, pre- and post-evaluation is not. On-line environments can allow flexible and 
just-in-time feedback loops that professors must strategically and creatively use to learn more about their 
competence in on-line course design. Eeedback on interaction with content, peers and instructors, accessibility of 
resources, user interface, and communication mediums, for example, would allow the instructors to better meet the 
learning needs of their students. 



Purpose 

The purpose of this study was to describe students’ perceptions of on-line consumer science and merchandising 
course design. The following three questions guided this study; 

1 . What are students’ perceptions of their interaction with course content and instructors in an on-line 
environment? 

2. What are students’ perceptions of accessibility of resources in an on-line environment? 

3. What are students’ perceptions of on-line course engagement? 

Research Design 

This study focused on pre- and post-evaluations of nine sections of seven on-line courses in an 
undergraduate consumer science and merchandising program. A total of 191 students were enrolled in the nine 
sections. The following describes the sample and population, instrument, data collection procedures, and data 
analysis. 

Sample and Population 

To answer the proposed research questions a convenience sample of 191 students enrolled in nine on-line 
sections of seven courses in the Department of Human Development and Consumer Sciences was selected. The 
sections used for this investigation included spring, summer and fall terms for 2002. The population was students 
enrolled in on-line courses leading to a Bachelor of Science in Consumer Science and Merchandising. The 
university in which this study was conducted is a large, state-supported urban institution of approximately 33,000 
students. One of the sections included was a freshman level course; three of the sections included were sophomore 
level courses; three of the sections were junior level courses; and two of the sections were senior level courses. 



Instruments 

Pre- and post-evaluation instruments were designed specifically for this project. The pre-course evaluation 
contained 16 closed-ended format questions. Seven of the questions were demographic in nature including items 
requesting input on gender, class standing, major, ethnicity, employment, and course load. The other nine questions 
asked for students’ perceptions of their anticipations regarding such factors as interactions, accessibility, and design 
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of instruction. Student perceptions were collected using a Likert-type scale of strongly agree, agree, disagree, and 
strongly disagree. 

The post-course evaluation included 23 closed-ended format questions. The first seven questions were 
identical to the demographic questions included on the pre-course tool. The additional 16 closed-ended questions 
solicited students’ perceptions of their course experience. Seven of those items were similar to items asked on the 
pre-experience survey. Again, factors such as interactions, accessibility, and design of instruction were included as 
well as items designed to assess overall satisfaction with the course. 

Data Collection 

Using the pre-course instrument, data was first collected at the beginning of the semester. The instrument 
was placed on the course website and students were directed to complete it. Then, at the end of the term, students 
were given access to the post-course instrument and instructed to complete it. Students completed the instruments at 
their own worksite and at their convenience. 



Findings 

Data Analysis 

Preliminary analysis of the pre- and post-experience data was accomplished via the quiz and survey 
capability of the course software. The delivery medium used for these courses was WebCT. Following initial 
review, the data from the closed-ended questionnaire items was evaluated by univariate frequency distributions. For 
these closed-ended items, response categories agree and strongly agree, and disagree and strongly disagree were 
collapsed to become agree and disagree. This was done to enable a generalized view of students’ perceptions. 

Comparing Pre- and Post-Experience Perceptions. 

Following initial analyses, the seven similar questionnaire items from both the pre- and post-course 
instruments were compared. Percentages were calculated to aid in the comparison. Then, to enable further 
examination and interpretation, the data was clustered by content areas that corresponded to the three research 
questions. The Paired Proportion Test for Dependent Samples was used to test for significant differences between 
anticipated and experienced scores. 



Results 

Respondents were predominately female (73.30%) and college seniors (59.16%). Majors enrolled in these 
consumer science and merchandising courses included Consumer Science and Merchandising (24.61%), Human 
Development and Family Studies (10.47%), Human Nutrition and Foods (1 1.52%) and other (49.74%). Ethnic 
distribution included white (39.79%), African American 28.80%), Asian American (16.23%), Hispanic (10.47%) 
and International (2.09%). Students were employed more than 20 hours per week (71.73%) and were enrolled in 10 
or more credit hours (67.01%) (see Table 1). 

Research Question #1: Students’ perceptions of their interactions with course content and instructors. 

Findings related to Research Question #1 indicated that students anticipated (83.77%) and experienced 
(85.18%) ease in paying attention to the instructor’s materials on the computer monitor. They anticipated (94.24%) 
and experienced (79.89%) the ability of the instructor to pay attention to students on the web. Anticipation and 
experience with other interaction aspects was also positive; ease of reading graphics and visuals (positive 
anticipation, 92.14%; positive experience 91.54%), ability to pay as much attention as in a regular course (positive 
anticipation, 87.43%; positive experience, 73.55%), instructor answering questions (positive anticipation, 92.67%; 
positive experience, 83.02%). Similarly, while not included in the pre-experience instrument, reflective summative 
responses also indicated positive perceptions of interaction aspects of the on-line courses. Class examples and 
activities were viewed as helping the student to learn (78.31%) and students felt that the faculty seemed 
knowledgeable about all course topics (78.84%). 

To analyze the differences observed in anticipated and experienced percentage scores the Paired Proportion 
Test for Dependent Samples was run on three pairs of data with differences large enough to merit analysis. These 
pairs included: “Ability of instructor to pay attention to students” (94.24% anticipated, 79.89% experienced), 
“Ability to pay as much attention as in a regular course” (87.43% anticipated, 73.55% experienced), and “Instructor 
answers question” (92.67& anticipated, 83.02% experienced). Analysis showed that only at the .1 level were 
percentages for the items “Ability of instructor to pay attention to students” and “Ability to pay as much attention as 
in a regular course” significantly different. Analysis of the item “Instructor answers questions” did not show 
significant differences in percentage values (see Table 2). 
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Research Question #2: Students’ perceptions of accessibility of resources. 

Relating to Research Question #2 findings showed that students both anticipated (91.63%) and experienced 
(90.48%) adequate access to resources needed for the class. Support personnel were also seen as readily accessible 
(anticipated, 77.49% & experienced, 53.44%) (see Table 3). (Approximately 30% of the post-experience 
respondents thought this question did not apply. Thus, tests for significance were not performed.) 

Research Question #3: Students ’ perceptions of their engagement with on-line courses. 

Only one of the instrument items solicited comparative pre- and post-course data regarding perceptions of on-line 
course design. The ability of students to pay as much attention in the on-line course as in a regular course received 
ratings of 87.43% (anticipated) and 73.55% (experienced). (For implications note that more experienced students 
did not feel the question applied). 

Students’ General Satisfaction. 

Following students’ on-line experience, several additional questions were asked and tabulated. Eighty-one 
percent (81.48%) agreed that the flexibility of using on-line technology helped them learn the course content. Only 
12.69% disagreed. Course organization was also assessed. About 75% of the students thought their course was well 
organized. Just under a quarter (22.75%), however, disagreed that their course was well organized. Testing was 
also evaluated with 81.86% reporting that testing was handled appropriately. Only 7.94% disagreed with the 
handling of testing. 

When asked whether the course content met students’ expectations, 79.37% responded positively while 
only 14.82% responded negatively. Additionally, 77.78% of the students stated that they would tell their friends to 
take an on-line merchandising course while only 12.7% said they would not. Similarly, 73.02% stated that they 
would take another on-line merchandising class themselves while 17.46 said they would not. 

More globally, student anticipation and perception of their overall satisfaction with the on-line experience 
were favorable. Pre-experience evaluation scores were nearly 90% favorable with only 6.81% unfavorable. 
Correspondingly, post-experience evaluation scores were nearly 80% favorable with 15.88% indicating 
dissatisfaction (see Table 4). In general, students’ anticipation scores were somewhat higher than experience scores. 
Reasons for this difference merit further investigation including whether the discrepancy may be due to unrealistic 
expectations. Lower experience scores may reflect an adjustment to the reality that on-line courses are not easier 
than face-to-face courses rather than that courses have poorly designed components. 

Implications 

In general, these findings indicate that students held positive perceptions of on-line courses. This was true 
both as they anticipated taking an on-line course and as they had experience with that course. This perhaps suggests, 
first, that there are not insurmountable barriers for students to taking on-line courses. Second, it suggests that for 
these specific courses the experience was generally positive. From these two factors it may be argued that on-line 
coursework may be a positive, viable option for delivering course content to students. 

Interactions 

From the data generated by the cluster of questions related to students’ perceptions of their interactions 
with course content and instructors, it may be implied that students had positive interactions. Student responses 
indicated that they were able to pay attention to course materials. In fact their summative responses were even 
slightly more positive than their pre -experience responses. Additionally, most expressed that they were as able to 
pay attention in the on-line course as well as they had in face-to-face courses they had taken. This implies that 
students did not feel that it was necessary to have physical or face-to-face interactions with an instructor or the 
course content in order to pay attention. This finding may alleviate faculty fears relating to student inattention and 
free them to concentrate on the development of excellent on-line strategies. 

Predominately, students also anticipated and experienced the ability of the instructor to pay attention to 
them in the on-line course. This suggests that faculty appear ready to help students and to respond to their needs. A 
slight drop in ratings between anticipation and experience, while still positive, indicates that there may be an 
opportunity for faculty to be more responsive to students and perhaps to incorporate more interaction between 
students and instructors into the course. This may also apply to the way in which faculty answer student questions. 
Strategies to encourage and facilitate effective faculty-student communication could be developed. These could 
include course format design, curricular solutions, and faculty training. 

Student’s positive responses to other interaction issues such as ease of reading graphics and visuals and 
class examples that helped students to learn suggest that current course design elements are meeting students needs. 
This finding has merit because it suggests that present efforts to develop effective courses are being successful. It 



369 




does not, however, imply that current strategies and techniques will remain effective or even adequate. Both 
students’ levels of sophistication and changing technologies will demand vigilant attention to the development of 
ever-superior graphics, visuals, curricular elements, and the technology to deliver them. 

Finally, with regard to interaction issues, this study reports that students felt that the faculty was 
knowledgeable about all course topics. This, too, is a positive finding. It reflects student confidence in the 
knowledge held by the instructor and may, perhaps tenuously, be extrapolated to suggest that faculty were effective 
in sharing that knowledge or in creating a learning environment where students could develop their own knowledge 
base. As such, the ability of faculty to couple their own knowledge with the capacity of on-line delivery strategies 
to enable students to have productive interactions with subject matter is a positive finding. This finding is well 
worth exploring, as technology advancements will continue to challenge this potential. 

Accessibility 

The cluster of questions regarding students’ perceptions of the accessibility of resources had two 
components. First, responses indicated that students both anticipated and experienced adequate access to the 
resources they needed for the class. While the question itself is not specific in the type of resources referred to, 
perhaps the message of the responses is that resources can be provided on-line. Creative instructional designers, 
including faculty, have the capability to decide what resources students will need for optimal learning and then to 
provide venues for students to obtain them. This is important because with time, resource needs change. Yet, these 
findings imply that securing access to resources for students is, indeed, possible. Increasingly, technology may 
prove an excellent ally in the procurement of course materials for student access. 

Second, more than half of the students in this study reported that support personnel were readily accessible. 
While this finding is positive, lower positive response rates to this item in comparison to other survey items causes 
concern. Only about three-quarters of the students anticipated that support personnel would be readily accessible. 
This indicates some apprehension on their part. Even more drastic is that only just over half of the students 
experienced readily accessible support personnel. This may indicate less than ideal service associated with these 
particular courses. However, the low satisfaction number may be because almost a third of the respondents did not 
answer this question or did not think that the question applied to them. In those cases it is possible that they, simply, 
had not felt the need for support personnel and therefore were unaware of whether they were available. In either 
case, it seems imperative that for students who need support, particularly technical support in the case of the use of 
new technologies such as on-line instruction, that support be readily accessible. This can be accomplished by 
decisions to hire support persons, instructing students on how to request support, and training faculty and staff in 
how to provide quality and timely support. 



Learning Engagement 

One component of on-line course design evaluated was the ability of students to pay as much attention in 
the on-line course as in a regular course. Once again, while student responses were positive, a slight drop from 
anticipation scores to experienced scores may be cause for concern. A question asking whether there were elements 
of the course design that did not meet students’ engagement expectations is an appropriate addition to the summative 
evaluation. Other factors such as what are students’ expectations, what features captivate and/or hold students’ 
interest, etc. could also be included in future evaluations. 

The findings associated with this study can provide input to future investigations. These students felt that 
the flexibility of using on-line technology helped them learn the course content. Further research could look at why 
flexibility was advantageous. Similarly, testing issues could be explored. This study made no attempt to learn 
which testing methods were favored but learned only that students were comfortable with the handling of testing. It 
would be useful to compare further the types of testing strategies used in comparison to student satisfaction and how 
well they measured student learning and application. Course organization could also be examined. Consideration 
could be given to elements such as course structure, instructions to students, orientation materials, etc. 

Finally, questions such as would students tell their friends to take an on-line course, would they themselves 
take another on-line course, and their general satisfaction with the on-line experience have strong implications for 
future study and application. This study highlights many opportunities for investigation of on-line learning. 
Strategies, priorities, efficiencies, and expectations are all fertile fields. However, perhaps the greatest impact of this 
study is the clear recognition that these students perceived on-line learning positively. This can provide incentive 
for educators and theorists to continue to push the horizons of this new learning environment to create ever- 
improving modes of instruction perhaps based on these early (successful) endeavors with on-line learning. 
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Table 1 Student Demographics 



101 . 



Characteristic 



Percentage 



College major 



Consumer Science and Merchandising 



24.61 



Human Development and Eamily Studies 



10.47 



Human Nutrition and Poods 



11.52 



Other 



49.74 



Ethnicity 



White 



39.79 



African American 



28.80 
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Asian American 



16.23 



Hispanic 10.47 

International 2.09 

Employment status 

More than 20 hours per week 71.73 

Enrollment status 

Enrolled in 10 or more credit hours 67.01 

N= 191 
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Table 2 Student Perceptions of Interactions with Course Content and Instructors 

Characteristic Percentage 

Anticipated Experienced P V alue 



Ease in paying attention to 
instructors’ materials 


83.77 


85.18 




Ability of instructor to pay 
attention to students 


94.24 


79.89 


.067** 


Ease of reading graphics and 
visuals 


92.14 


91.54 




Ability to pay as much attention 
as in a regular course 


87.43 


73.55 


.066** 


Instructor answers questions 


92.67 


83.02 


.158 


Examples and activities helped 
the student to learn 


* 


78.31 




Faculty seemed knowledgeable 
about all topics 


* 


78.84 





N= 191 

* Factor included only on experienced instrument 

*Paired Proportion Test for Dependent Samples found these differences to be significant at the .1 level only. 
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Table 3 Student Perceptions of Accessibility of Resources 

Characteristic Percentage 

Anticipated Experienced 

Adequate access to resources 91.63 90.48 

Support personnel were readily accessible 77.49 53.44* 

N= 191 

* Approximately 30% of the experienced respondents thought this item did not apply. 

Table 4 Students’ General Satisfaction 

Characteristic Percentage Agree 

Flexibility of technology helped them learn course content 81.48 

Course was well organized 75.66 

Testing was handled appropriately 81.86 

Course met student’s expectations 79.37 

Student stated they would tell their friends to take an on-line course 77.78 

Student stated they would take another on-line course 73.02 

Student was satisfied, overall, with the class 79.9 

N= 191 
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The ’New Wave’ for Technology Integration in California Schools 



LeeAnnStone 
Teacherr^c/z Connection 

Introduction 

With the completion of the Digital High School Program (DHS) in 2002, California's high schools have 
experienced the first major wave of effort aimed at building their technology capacity. As a result of the program, 
student-to-computer ratios among California high schools were reduced from 16:1 (Milken Exchange, 1999) to to 
5.3:1 across all California schools by 2002 (California Department of Education, 2002). Yet despite this extensive 
infusion of equipment and technology infrastructure, the same degree of impact has not occurred in the area of 
utilization of these resources. So common is this phenomenon that, several participants in the study named it for us: 
Technology rich, utilization poor. This paper reports findings on a study of barriers to technology integration in 
eight selected high schools in Southern California, and reports recommendations for the next wave of technology 
integration efforts. 



The Research Design 

The research upon which this report is based sought to identify the continuing barriers to technology 
integration now that the critical factor of availability has been addressed. Many other researchers have tackled this 
question as well (Borko & Putnam 1995; Brickner, 1995; Byrom, n.d.; Chu, 2000; Dias, 1999, 2000; Ertmer, 1999; 
Ertmer, Addison, Lane, Ross & Woods, 1999; Hadley & Sheingold, 1993; Hugo, 2000; Mehlinger, 1997; 
Montgomery 2000), most notably Sandholtz, Ringstaff & Dwyer (1997). While the findings of these earlier 
researchers provide a foundation upon which this study builds, they focused for the most part on a single variable — 
teachers and the issues they face both intrinsically and extrinsically — in their search for barriers. This study, while 
noting the critical role of the teacher in the integration process, employs a holistic lens that looks at a network of 
interrelated factors that can impact technology integration in schools. 

The central research question of this study is: What barriers to the utilization and integration of new 
technologies into teaching and learning are experienced and articulated by students, teachers, administrators, and 
staff, and expressed within documents, from eight selected Southern California high schools? The framework for 
answering this question is derived from Lemke and Coughlin's Technology in American schools: Seven dimensions 
for gauging progress (1998); these dimensions include: the learner, the learning environment, professional 
competency, planning and leadership, community connections, technology capacity, and accountability. A series of 
additional questions incorporating these dimensions were established to serve as objectives in answering the main 
research question. Live of the eight questions are the focus of this report, and are detailed in Eigure 1 . Employing 
this framework in the investigation of barriers to integration allows for a systemic exploration through multiple 
lenses, and provides an opportunity to build a comprehensive picture of the interrelationships among the myriad 
obstacles that present themselves in the course of technology integration within a given school. 

Eigure 1 : Sub-questions used to guide analysis of the data 



1 . The learner: What role — from the student’ s perspective — do any district, school, classroom, and/or 
lab policies and practices play in promoting or obstructing new technologies use? 

2. The learning environment: What are the conditions of access inside and outside the school setting, 
and in what ways do these conditions promote or obstruct teacher and student use of new technologies? 

3. Technology capacity: What are the conditions of the technology capacity of the school and classrooms, 
and what role does this capacity play in terms of student and teacher access, availability, and use? 

4. Professional competency: What do teachers express in terms of their professional knowledge, 
experience, and competencies related to new technologies for teaching and classroom 
management, and are there any patterns of facilitations or barriers that relate to their 
instructional practices? 

5. Planning and leadership: What is the state of technology planning and leadership within the school, 
and how does the coordination and communication of that planning facilitate or impede the 
implementation: of technolosv integration? 
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The data source for this secondary analysis comes from a study for the University of California Office of 
the President on access, availability, and utilization of new technologies in California high schools (Knobel, Stone & 
Warschauer, 2002). Table 1 outlines the type, counts and form of the data used in this secondary analysis. 

Table 1: Inventory of Data Type, Counts, and Forms 



Data type 


Quantities 


Forms 


Teacher interviews 


59 


Audiotapes, filed notes & transcripts (both 
electronic and hard copies) 


Administrator & staff interviews 


15 


Student focus groups 


9 


Classroom observations 


79 


Student questionnaires 


1052 


Original questionnaires & FileMaker Pro database 
of responses 


Artifacts & documents 


122 


Copies of documents 



Three analytic techniques were employed with this data: (a) emergent pattern matching as applied in the 
ethnographic tradition (Fetterman,1998; Knobel & Lankshear, 2001; Stemler, 2001; Yin, 1994), (b) a priori pattern 
matching, and (c) counts and correlations. Emergent pattern matching allows for the identification of themes and 
patterns that may not have been previously identified, while a priori provides the researcher the opportunity to build 
upon the findings of previous researchers. Counts and correlations were employed to reveal any relationships among 
the demographic data, as well as triangulating data found through the pattern matching processes. 

Findings: The Barriers 

The Learner 

Student responses to survey questions — particularly the open-ended questions — and their comments from 
focus groups provided an opportunity to access information about technology implementation (or lack of it) across 
the school, via the students’ experiences with a wide range of courses and teachers. Such data is extremely sparse 
among the literature on the topic, first because the students' perspective has not been the focus of much research 
interest, and secondly — if the experience of this research project is any indication — due to the difficulty in getting 
access to students through district policy and Institutional Review Board processes. Yet the 1,082 student surveys, 
with their combination of closed- and open-ended questions, provide a wealth of information about students’ 
attitudes and sources of information about, and uses of new technologies. Additionally, the comments of the 49 
students who convened in 9 small group interviews provided a means to triangulate with patterns identified through 
the more extensive student survey data, as well as from other interviews and documents. Analysis of this type of 
qualitative data goes beyond the more commonly-applied quantitative approaches of equipment, student, teacher, 
and infrastructure counts, and appears to be the next wave of research into the topic. Indeed, in addition to current 
calls for deeper exploration into new technologies integration, which include in-depth qualitative approaches 
(Knobel, Stone & Warschauer, 2002), California's Technology Assistance Program (CTAP) has begun including 
some of these qualitative elements in its annual survey of California schools. 

From whom do the students learn? Across multiple variables within this study — across schools of varying 
socio-economic status, and across levels of home ownership of computers — students reported learning more about 
computers from themselves than from any other source. With a few exceptions, teachers ranked fourth after self, 
family, and friends in terms of influence on student learning as it relates to new technologies use. The question 
eliciting this data, "Who has taught you the most about how to use a computer?", consisted of a five-point Likert 
scale, with 1 indicating “Not much’’ learning from the source, to 5 indicating “A lot’’. Table 2 provides a summary 
of the contrasts between the high and low responses among the data. To arrive at the schema below, the number of 
responses in the 1 and 2 range (below the midrange) were counted and compared with the total of those responses 
above the mid range (i.e. 4 and 5). The data can be read thus: 

-n- There is a 100% difference or more between the number of responses at the high end of the scale (“I 
learned a lot from this source’’) and number of responses at the low end (“I didn’t learn much’’). 

+ There are more responses at the high end of the scale than at the low. 
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= The difference between number of responses at either end of the scale are equivalent or within 4% of the 

total count for that category (e.g. 43 low and 41 high equals a 2.3% difference between the counts, placing 
it in the roughly equivalent category). 

There are more responses at the low end of the scale than at the high. 

There is a 100% difference or more between the number of responses at the low end (“I didn’t learn much”) 
and number of responses at the high end of the scale (“I learned a lot from this source”). 

Table 2: Student Self-Reported Sources for Learning Computer Use 



Learn about computers 
from; 


Bergenia 


Cassia 


Dalea 


Erica 


Escallonia 


Kalmia 


Salix 


Vallota 


Self 


-H- 


-H- 


-H- 


-H- 


-H- 


-H- 


-H- 


-H- 


Friends 


= 


+ 


+ 


- 


-H- 


+ 


+ 


+ 


Family 


+ 


+ 


+ 


= 


+ 


= 


+ 


+ 


Teachers 


+ 


— 


— 


= 


+ 


— 


- 


— 


School librarian 


- 


- 


- 


- 


- 


- 


- 


- 



When schools are ranked in terms of rates of home ownership and that ranking is compared with student 
ratings of the impact teachers have on their computer learning, an inverse relationship between the two emerges. 

That is, students at schools with lower home ownership rates rank teachers as more influential in their learning about 
computers than do students at schools where the rate of home ownership of computers is higher (see Table 3). 
Ironically, however, teachers at schools with lower rates of student home ownership of computers reported assigning 
less computer-involved work both at school and at home than teachers at schools with higher rates of home 
ownership. Thus, those students who rated their teacher’s influence as higher actually had fewer opportunities to be 
involved with technology-based learning. Teacher and administrator interviews at schools with lower rates of home 
ownership revealed that their reduced assignment of technology-based homework was an explicit teacher or school 
policy designed as a means to not disadvantage students who may not have access to computers at home. Without 
exception, teachers and administrators at these schools estimated that only 50% of their students had home access. 
Not only did students report significantly higher rates (Table 3), but many who do not have home computers stated 
that they had a variety of options for access (including friends, family and libraries), and that doing so is not a 
problem in terms of completing assignments. Thus, students who most looked forward to learning from their 
teachers, had the least opportunity to do so due to policy based upon faulty (yet honorable) assumptions. 

Table : Relationship between Home Ownership of Computers and Teacher Influence on Computer Learning 





Low-SES schools 


Higher- SES schools 


Escallonia 


Bergenia 


Erica 


Salix 


Kalmia 


Dalea 


Cassia 


Vallota 


Learn from teacher 


+ 


+ 


= 


- 


- 


- 


- 


- 


Home ownership rates 


74% 


75% 


83% 


92% 


94% 


98% 


99% 


100% 



The Learning Environment and Technology Capacity 

The capacity and conditions of access within the learning environment weave around one another as factors 
that influence to what extent both teachers and students utilize new technologies. Capacity refers to the 
infrastructure of networking and support, as well as actual inventories of equipment within a given school 
environment. Until recently, capacity has been the primary focus of research in the area of technology integration. 
Annual counts of computers and internet connectivity (e.g. CTAP data) have represented the key criteria for 
determining degree of technology integration within California schools. However, such counts alone provide only a 
glimpse of the overall structural framework of new technologies within a school, and provide no real information 
about how that infrastructure is accessed, nor what is done with it (Knobel, Stone & Warschauer, 2002). 

Table 4 provides a summary of student- to-computer ratios at the schools participating in this study. If all 
that were available were this data, one might surmise that the schools with the lower student-to-computer ratios had, 
perhaps, the most extensive and integrated uses of new technologies. This, as the current research found, is certainly 
not the case. In fact, Salix High School, with the lowest ratio among all participating schools, also had the lowest 
level of observed technology use. Many factors contribute to rich usage, beginning with how easily, or with how 
much difficulty, technology can be implemented within the instructional and learning processes. 
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Table 4: Student-to-Computer Ratios at Participating Schools 





Salix 


Cassia 


Escallonia 


Kalmia 


Dalea 


Bergenia 


Erica 


Vallota 


Computers for instruction 


600 


667 


327 


541 


688 


680 


506 


300 


Students per computer 


3.3 


3.7 


3.7 


4.0 


4.2 


4.5 


5.7 


7.8 


Internet connected computers 


600 


453 


327 


357 


459 


321 


506 


300 


DHS start year 


1999 


2000 


1999 


1999 


1998 


2001 


2000 


2001 



Note There is no evident correlation between student per computer ratios and low- or high-SES status or between 
DHS start date and student computer ratios. 



Teacher access. Although Digital High School grants require at least one internet-connected computer per 
instructional space, the reality of physical growth and layout at many campuses results in schools not meeting this 
requirement. Capacity in the form of internet connectivity is impacted negatively by the ongoing ad hoc growth of 
campuses via portable classrooms (Bergenia, Dalea, Kalmia, Salix), as well as scheduled renovations (Salix), and 
major expansions (Bergenia). 60% of Bergenia’s classroom spaces are housed in portable buildings with no internet 
connectivity, and at Kalmia, 29 out of the 91 classrooms are added portable facilities. Not only does Salix have a 
village of portable classrooms, but due to a major renovation project, full departmental wings of classrooms are 
currently disconnected from internet access; one-third of the campus has no internet connectivity. Two teachers at 
Bergenia mentioned that last year they had internet connectivity and no computer, and this year the reverse. Such 
capacity issues directly impact teacher’s abilities to plan for and implement new technologies use. 

Thus, teachers in the study had access that ranged from only one stand-alone computer to 25 networked and 
internet-connected computer stations in their classrooms. The range of access by teachers and their classes to 
computer labs was equally as wide. At Bergenia, teachers are not even afforded the opportunity to schedule use of 
the lab to fit into their instructional needs. Rather, they are assigned lab times by a non-teaching, part-time lab 
monitor. Teachers can attempt to swap assigned times with other teachers to fit use into their curricular needs. On 
the other end of the spectrum, English teachers at Dalea routinely, and even without advance notice, share and 
exchange classroom spaces depending on need. 

Security procedures, in particular security of the system from hacking and/or posting of inappropriate 
materials (whether due to copyright or content), create access issues for teachers. Many of the complaints from 
teachers and technology coordinators, such as the following by the Media Resource Specialist at Kalmia, included 
references to cumbersome and convoluted district policies: 

[T]hey [the Districtjwant us to have all of our web pages, the school web page, as well as the library web 
page on the District server for protection. My web page was originally on the District server, but they put 
so many obstacles in my way and hindrances that I decided not to put it there anymore. Originally, to put it 
up there, you had to learn the special program that they had. . .stupid program that was very difficult to 
learn, and the only one that knew how to do it was one person on campus. That was me and I had to learn 
from them and they just told me how. Anybody else needs anything else on my web page, you gotta come 
to me and get the password and I have to upload it for ‘em.... Any time I wanted something on the internet, I 
had to go to my principal and get approval. And then I would email the guy at the District and he would 
email my principal to get approval. The principal emailed him, "Okay, I checked it out. It’s okay to put it 
out.", and he would put it out. Weeks would go by sometimes. 

As a result, the Media Specialist at Kalmia made a unilateral decision to host the school web site at his own expense. 
Media Specialists at at least two other schools in the study made similar decisions. Not only did they feel that they 
could maintain more consistent uptime on their systems, but they found it much easier and quicker to make regular, 
even daily, changes to their web pages. This is an increasingly important factor if web pages are to function as 
effective providers of up-to-date information and communications. 

Physical plan. Data from this study illustrated how the physical layout of campuses can obstruct or provide 
affordances for collaboration, which in turn can facilitate advancement of teacher professional capacity and 
integrative use of technology. At Dalea High School, departments are housed mostly in distinctive wings, each of 
which provides common meeting spaces housing desks, bookshelves, and kitchenettes for teachers. In the English 
Department, for example, teachers routinely, and even without advance notice, share and exchange classroom spaces 
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depending on a teacher’s and his or her class needs for the day. Indeed, the other schools where high levels of 
regular teacher collaboration were observed had similar architectural layouts (Escallonia and Cassia). These are also 
the schools that demonstrated the most frequent and integrative uses of technology within their curricula. 

Bergenia presents a striking contrast to the compact and collaborative environment of Dalea, Cassia and 
Escallonia. One trailer village (housing science classes) sits on the far end of the football field, and access to those 
rooms is locked off by a 10-foot high fence between class periods. Another crop of portable classrooms occupies the 
basketball and volleyball courts. Teachers at the campus cannot tell visitors how to find rooms outside of their 
immediate vicinity. The campus is so large both in terms of student population (over 3000 students) and in 
geographic sprawl, that campus communications in general are strained. The Assistant Principal related the situation 
in this way: 

When I went into administration, and people said where did you come from, and I said Room 7b, and they 
said, “You’ve been here at the school?’ They didn’t know I was at the school. It’s like a small city. 

We’ve just had this tremendous, between growth and retirement, big turnover in our staff. So people don’t 
know, “Where can I find Mr. so and so. . .7” “I don’t know. I’ve never heard of that person”. I’ve had 
people say to me, “Who was that you were talking to? Does he work here?” “Yes”, because of the way it’s 
laid out. And we don’t have a lot of staff meetings. The principal just didn’t believe in calling meetings. So 
people don’t see each other. The size of the campus is a barrier because it’s so large that networking is an 
issue. 

The physical layout of schools, therefore can create both barriers to physical networking (i.e. inability to provide 
internet cabling) as well as barriers to the collegial networking that allows teachers to share resources and ideas, and 
to build upon one another's practice. 

Technical support. Concerns about the unpredictability of staffing funds plagued technology coordinators 
and administrators across the data set. The Assistant Principal at Cassia High School summarized it thus: 

What’s going to really happen to technology cause it scares me to no end. The district is basically saying 
that, “It breaks, you’re going to fix it, pay for it.” That’s great as long as the funds are there, but you’re not 
necessarily guaranteed those funds being there forever. And there are places you can go to see where all the 
leftover computers are. There’s piles of them here and there that are just categorized for parts. ..We need 
more support, but the fear of funding cuts probably really diminished that. ..Until we know there’s going to 
be money in the future, we kind of cut back on purpose for putting another technician in the school site to 
do a lot of the maintenance so that we can eventually free up more teachers to do more training on an 
ongoing basis. That’s what was really going to happen. 

A few schools, such as Vallota, Dalea, and Erica, had begun the new technologies integration process long 
before the DHS push. As a result, the network infrasture was mature and the support team had more experience in 
handling issues related to that support. At most of the schools in the study, however, such was not the case. DHS 
funding represented one of the biggest grants the schools had ever received, and so many went from having a dozen 
stand-alone computers throughout the campus to hundreds of networked ones. The technology capacity at Kalmia 
High School, for example, grew from 20-30 Apple He computers and a Mac Classic keyboarding lab, to over 500 
computers in a couple of years. Other schools in the district went through similar changes, and yet, as the Media 
Specialist at Kalmia explains, the staffing infrastructure did not grow accordingly. 

There were [only] two guys for PC and two guys for Mac. And then there were three guys for AV stuff [at 
the District offices] . . .that were available to come out and do networking stuff. . .And then there was the 
Network Administrator and the Director of Information Systems people in that office downtown. But 
people who actually could come out and work were two PC, two Mac, and really two guys who would 
actually come out our office for AV stuff. And that was it. And that was for eight high schools, eight junior 
highs, three adult ed campuses, the Academy, and the District Administration building. And again, you got 
a District that went from, I don’t know, district-wide, how many computers, but now you’re talking about 
thousands of computers around here in the District. And it happened over about a two-year period. 



In response to the need for support staff that includes both technical and pedagogical support, as well as 
planning units, Dalea High School has established for itself over time what can be viewed as a tapestry of 
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interconnected support networks. Through this network, the campus has created a cadre of technology-competent 
teachers who themselves can “train up” as the Media Specialist phrases it, other teachers on campus. This scenario, 
however, highlights the situation that all campuses face to some degree: that necessary support comes in the form of 
committed teachers and other campus personnel who add, often without additional compensation, to their already 
multiple assignments. Schools that have not established networks of technology support teams — and that is most of 
the schools in this study — are at extreme disadvantage in meeting the support needs of the school in light of the 
shortage of funding support, in particular, the deferment of TSST funds. 

Class schedules. Becker's (2000) finding of more frequent use of computers during class time by teachers 
on block schedules than those on traditional schedules was observed in this study as well. Of the participating 
schools. Cassia, Dalea, Erica, and Vallota all function on variations of block schedules. In addition to simply 
providing longer periods to do extended activities, block schedules provide affordances for accommodating the time 
consumption that using new technologies often entail. In an interview, the technical coordinator at Dalea High (a 
school that has not implemented a block schedule) explains the barrier that short class periods create: 

The short period is a barrier. In terms of using technology... particularly if you’ve got 200 kids, 250 kids 
logging into the Mac server in the first 10 minutes, and the same thing on the PC server. If you’ve got that 
many kids logging in on the PC server, sorry, there goes your first five, ten minutes, maybe. If there’s 
something wrong. And if the server goes down, sorry, there goes 15, 20 minutes and by the time you got 
any work down, it’s time to save. ...So there’s a lot of time when teachers and students will lose a whole 
period if there’s even a minor technical difficulty just because by the time they get a hold of us, then it gets 
taken care of - I mean, we have what in industry would be a bitchen turn around time, you know, if you 
lose a half hour, who cares. Go make your coffee. Who cares. You go back to work. You lose a half hour in 
a 55-minute period, your day’s shot. Especially if it takes 10, 15 minutes to talk to the kids and then get 
them in the lab. 

Vallota’ s block schedule is unique in providing 45 -minute office hours twice a week. During these office 
hours, students are permitted to meet with teachers and/or work on individual or group assignments, in the computer 
labs if necessary. Although these office hours were not designed specifically to provide students on-campus 
computer access time, teachers and administrators at Vallota alike mention the ways in which these periods 
contribute to student access. 

Student access. Access means more than simply whether can one get to a given piece of equipment; it also 
relates to once there, what one can do with it. At Vallota students are not allowed to use school facilities for personal 
email. At Bergenia, students are not allowed to use two of the labs for any independent work of any type. Other 
restrictive policies among the study schools include cumbersome and time-consuming check-in and check-out 
policies that discourage student use, prohibitions on students helping other students at their computer, and limited 
distribution of computers throughout the school. On the other end of this range of student access issues is Erica High 
School, where the administration has an open policy allowing students to explore new technologies resources on 
their own terms. A snapshot of what such access looks like at Erica follows in Eigure 2. While all of these schools 
have limited computing resources, each has made different decisions regarding allocation of access to these 
resources by students. Erica, with the second-highest student-to-computer ratio, has nevertheless made the decision 
to extend access for other than focused academic activities, a decision that allows students to explore ways in which 
technology relates to their own interests and daily lives. 

Professional Competency 

The data from this study disproves the popularly expressed belief that older teachers are generally 
reluctant users of technology. It is also popularly assumed that younger, less tenured teachers generally have used 
computers more in their own school and teacher training experiences than tenured teachers for whom personal 
computers did not exist in their school days. These assumptions were disconfirmed by the data from this study. On 
the contrary, a large percentage of participating teachers had both significant numbers of years of teaching 
experience (averaging approximately 17 years) as well as placement at their current school (averaging 13 years). 

37% of teachers participating in this study have taught for more than 20 years, with eight of those having 30 years or 
more of teaching experience. This finding coincides with other studies that have found that teacher’s uses of 
classroom technology evolve as they gain experience (Ertmer, et.ak, 1999; Hadley & Sheingold, 1993; OTA, 1995). 
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Figure 2: Learner Access at Erica High School 

A dozen or more students are lined up waiting for a computer to be vacated as others, who managed to arrive 
only seconds earlier than these, are already deep into their computer activities. The lab is quiet — not as a 
result of regulations — but as a consequence of the students’ interest in the sites they are at. A scan of the 
monitors reveals that few of these students are working on school assignments. One student is checking out 
the World Taekwondo Federation, another a wrestling site, several students are at games and music pages 
(eZone games and KROQ among them), while a car sales site and an online clothing page occupy two other 
students’ attention. Six of the 23 students are logged onto the Erica High School web page, although some 
perhaps for social information (games schedules, homecoming activities) rather than schoolwork. Of these 23 
students, 17 are young men and only a quarter are young women. 

The labs are open to the students to use on a first-come, first-serve basis regardless of their purpose in using 
them. This contrasts significantly with policy at other schools in the study where students are warned through 
many modes that computers are for schoolwork only. The Principal, Assistant Principal and Librarian at Erica 
all concur that “letting them play” is ultimately a productive route towards academic integration. 

Adapted from; Knobel, Stone & Warschauer, 2002 



Mehlinger (1997) concluded that even when teachers have easy access to computers and plenty of training, it takes 
at least three years before they feel comfortable enough to begin thinking instinctively about how to incorporate 
these into their lessons. Hadley and Sheingold’s (1993) data suggested an even longer time of 5 to 6 years. Thus, the 
disadvantage, as the data from this study illustrated, actually lies in being a novice teacher. 

A comparison of novice teachers to more experienced ones within the study highlight the role that 
experience plays in allowing teachers to venture into new teaching domains. Meskill, C., Mossop, J., DiAngelo, S., 

& Pasquale, R. (2002) conclude that: 

experienced teachers who use technologies effectively with their students can be broadly characterized as 
viewing technology as a means rather than an ends to learning (Garner & Gillingham, 1996), that they see 
themselves in a mainly advisory role (Meskill, Mossop, & Bates, 1999; Norton & Gonzales, 1998), and that 
as professionals they welcome newness and variety, and continually seek to expand their repertoires (Berg, 
Benz, Lasley, & Raisch, 1998). Novices, on the other hand, feel they must first and foremost master the 
routines and rituals of new contexts (Kagan, 1992). When these contexts include mandated technologies 
use, there is risk of non-reflective appropriation, (pp. 46-47) 

Differences between the discourse of novice versus experienced teachers from the data of this study confirms this 
finding. Here is what a novice, not-yet-credentialed Science teacher from Bergenia High School has to say about 
integration of new technologies resources into his curriculum: 

[I]f I spend a couple of days or one day a month in the computer lab, then that’s one day I’m not covering 
the content, we’re not doing a lab, it doesn’t count as a lab. For UC purposes, we have to be doing 40 % 
labs and that’s a really tough one to pull and so it takes me out. It’s not a lab day, they’re not working on 
the . . .or whatever. . . a writing. It’ s a technology thing, which is useful for a job, but it takes me away from 
doing other stuff. And so I’ve never in my four years taken them into the computer lab. 

Yet a more experienced science teacher from Erica addresses the same issue in a very different way: 

We have so much breadth that needs to be covered and so you don’t have that much time to go in depth for 
each topic. I don’t know if it’s just individual to our subject, I mean, that could be true for any subject, I 
think. And really, that’s where it comes down to determining and making a personal decision: Okay, I 
think my students are going to get a lot more out of doing this activity and if I don’t get it covered this one 
topic as much as I should. . .you’re constantly thinking these things. There’s no way you can follow a book 
exactly the way it is, which is exactly what we have to cover. I mean, we have our guidelines to set the pace 
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and our curriculum, this is what should be covered, and one thing leads to another. But it’s never going to 

be the same curriculum. And I think that’s our job. We’ve been trained to be able to make those decisions. 

This is one way in which being a novice teacher can create barriers in the technology integration process. 
Another related factor lies in the level of networking and collaboration that a teacher is involved in. A Bergenia 
Science teacher (an as yet uncredentialed teacher), for example, complained in his interview about the absence of 
centralized repositories of information and materials related to internet-based resources for Science instruction tied 
directly to California’s standards. An experienced English teacher in an interview that immediately followed, on the 
other hand, effused about S.C.O.R.E., an online resource for electronic lessons designed around California’s 
curriculum standards (including Science standards). The difference therein lay in the rich networks of professional 
colleagues and information resources that the experienced teacher had developed over the years. As a novice teacher 
engaged in completing his course requirements during non-teaching time, the Science teacher had not had the time 
or opportunity to yet develop such collaborative networks. 

Planning and Leadership 

The role of technology. In her case studies, Barbara Means (1995) found that a deciding factor in successful 
implementation of new technologies was the creation of a coherent schoolwide approach to using technology in the 
core curricula for all students. The data in this study found that despite such detailed school-developed technology 
plans, there existed wide differences among teachers and administrators, and between teachers and students relative 
to the role of technology in the classroom. At Salix, for example, the DHS coordinator cited his long-term 
technology goals for the campus to be that “computers become an everyday part of the curriculum, not just in the 
computer classes [as they currently are].” However, observations of classes did not substantiate this goal. Most 
disturbing from the classroom observations was the finding that students’ acquisition of discrete technical skills, 
such as cutting and pasting, and creating and positioning graphic elements into a Word document, were taught and 
performed by students devoid of any meaningful content or context. Eurthermore, while the role of technology in 
non-core classes such as journalism, art, music, yearbook, and computer graphics is direct and explicit, it is more 
frequently seen as an extra element in the core courses of English, Science, Math, and Social Studies. Cassia High 
School’s assistant principal stated that technology is “not a critical status of the academic areas.” School staff must 
have a clear understanding of the role of technology within their curriculum before they can begin to communicate 
and apply these goals to student learning. At a number of the schools in the study these understandings have not 
been jointly communicated, there is little common discourse or understanding, and the stated aims in the Digital 
High School grant applications and/or technology plans are in opposition to observed practice. 

Communication. What are the results when there is no overall campus communication and leadership? The 
situation at Salix is one example. Another example comes from Bergenia where at least three different campus units 
work relatively independently of one another in providing for student access to lab computers. In this case, two of 
the three campus computer labs run by the Project Office had internet connectivity provided through DHS funds. 
However, as a result of legacy issues — the labs previously had not had such connectivity, and each was designated 
for a particular set of functions — students were not allowed to use the internet in these facilities. This situation 
lasted for 5 months into the academic year before being resolved. The only reason students were not allowed access 
was because they never had it before and no one had given a directive to the Office or the lab monitors that they 
could make such changes to policy. A break in communication is the key to the problems that occurred here, and 
that is a leadership issue. To be able to make decisions that best meet the overall vision and goals, teachers, staff, 
and administrators not only have to be aware of those, but they have to be able to act on them — or at least make 
recommendations — as well. Had that been the case at Bergenia, the lab monitors may perhaps have felt both the 
responsibility and the authority to bring up an issue that teachers and students had expressed to them. 

Such communication among all constituents facilitates other aspects of technology planning as well. Eor 
example, teachers at several schools complained about higher-level decisions that brought technologies that they did 
not really use to their rooms, or pulled cables to inaccessible or unusable locations in their classroom. To be 
effective, therefore, leadership and planning has to take place at multiple, interconnected levels simultaneously. 
Without such an approach, equipment and infrastructure is not necessarily deployed in a manner that affords the 
most effective usage. 



Conclusions and Recommendations 

With $1 billion already invested in the process through the DHS program, the state of California has an 
intrinsic interest in finding ways to reduce barriers and accelerate the integration of new technologies into school 
settings. No single variable alone can provide all the necessary remedies for the integration process. It takes a 
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systemic effort of many of these variables to make the changes necessary to facilitate the effective integration of 
new technologies into schools. Findings from this study point to several recommendations for the next wave of 
technology integration efforts. These recommendations include: 

1. New technologies need to be made not just available, but also accessible for a wide range of student use. If 
students indeed learn more on their own than from school sources, as the data in this study indicates, school 
student use policies need to provide students the opportunity to explore areas of their own interest, opening 
opportunities for learning, even if not directly related to their academic assignments. Furthermore, schools 
and teachers need to hold higher expectations of student technology access and use. By withholding 
technology-involved assignments, teachers may be actually further disadvantaging students in terms of 
developing their new technologies understandings and skills. 

2. Schools need to build upon a robust hardware and network infrastructure on-campus technical expertise 
and support, a cadre of technology-savvy teacher mentors, and administrative support that includes 
effective modes of communication among these parties. When one of these elements is weak or missing, 
the integration process appears to suffer significantly. 

3. Schools, districts and teacher education programs need to continue efforts to build networks of professional 
development support, particularly for novice teachers. A common characteristic of effective technology- 
using teachers is their involvement in professional networks of support both within and outside of their 
schools. Schools can build this capacity of involvement by establishing cadres of teacher technology 
mentors who can provide focused, as-needed assistance to their peers. Teacher education programs can 
introduce pre- and in-service teachers to larger networks of professional practice that can provide model 
lessons, discussions, and collegial connections. These resources are proven means to help teachers establish 
those relationships that will aid them in sorting through and making meaning of the plethora of information 
they will encounter as new and developing teachers. 

4. School leaderships need to establish shared technology visions that include a focus on technology-infused 
content learning, and communicate among both teachers and students. Most importantly, though, this vision 
must involve a focus on content learning in which technology is infused as part of the learning process, 
rather than, as at some of the schools in the study, oriented to technology as the product of the learning 
experience (Meskill et ah, 2002). 

5. Both class schedules and physical plans of schools need to be reconsidered. Block schedules provide time 
necessary for the student-centered, hands-on time that technology brings to the learning experience. Well- 
planned physical layout of schools contributes to teacher collaboration and professional growth. 

These recommendations can be distilled to the following: the first wave of integration focused on then technology 
core; the next wave of efforts must look at the myriad elements of the sphere of integration that surrounds that core 
(Figure 3). 

Figure 3: Sphere of Integration 
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Examining Instructional Design and Development of A Web-based Course: 

A Case Study 

Bude Su 

Introduction 

The growth of using Web-based instruction in higher education has been remarkable. Between fall 1995 
and 1997-1998, the percentage of higher education institutions offering distance education courses in the United 
States increased from 33 percent to 44 percent. The percentage of institutions using asynchronous Internet-based 
technologies nearly tripled, from 22 percent of institutions in 1995 to 60 percent of institutions in 1997-1998 
(National Center for Education Statistics [NCES], 1998). More and more distance students are taking Web-based 
courses instead of learning through live audio or television. 68% of the master's students who participated in 
distance education classes used the World Wide Web while 45% and 29% of the master’s student used live or 
prerecorded audio or television (NCES, 2002). 

Although the Web-based education is growing fast, educators are not necessarily ready to teach online 
courses. Instructors in higher education are often asked by the institutions to build online courses or to adapt some 
of their face-to-face courses to be delivered online. But are they ready to offer Web-based courses? Do they have 
enough references to refer to? According to National Center for Education Statistics report (2001), one of the 
greatest difficulties to teaching online was the lack of guidance on how to teach with technologies in their own 
situations. It seems clear to us that the instructional design field needs to have new and revised methods of 
instruction in order to take full advantage of the World Wide Web in education. In this study, through examining 
the instructional design process of a graduate level Web-based course, the researcher hopes to provide useful 
references for educators who may need to conduct a similar Web-based course in the near future. 

Research questions 

1. How did the instructors and course designers build the Web-based course? 

2. What major changes were made during course adaptation? And why? 

3. What are the students’ reactions to the course features? 

Conceptualization and literature review 

Instructional design is also known as instructional systems design (ISD). It is a systematic development of 
instruction using teaching and learning theories to ensure the quality of instruction. The instructional design process 
is concerned with “what process[es] a teacher or instructional designer should use to plan and prepare for the 
instruction’’ (Reigeluth, 1999). It is the process of combining analysis of learning needs and objectives with the 
design and development of a delivery system to meet those needs; it contains development of instructional materials 
and activities, and tests and evaluates all instruction and learner activities. 

Designing any type of instruction involves identifying the overall instructional goal and the corresponding 
theoretical and pedagogical strategies that will help accomplish that goal. When people are faced with developing an 
online course that needs to be delivered through a relatively new medium, such as World Wide Web, they tend to 
concentrate on the technological issues such as the capabilities of the medium instead of focusing on the 
instructional objectives, learner needs, and the specific task designs etc. (Richer, 1994; Bannan & Milheim, 1997). 
Littlejohn (2002) pointed out that there are several critical issues in instructional design in Web-based Higher 
Education that should be addressed to match the rapid shift towards online learning. 

Should be “Focusing on course outcomes rather than content" : Heavy content based course design without 
considering the possible learning outcomes may result in unengaging learning experience— “leading to 
surface learning”. Eor this reason, educators should focus on the achievement of learning outcomes when 
developing online courses. 

Should be “Placing dialogue and feedback central to course design”: Merely assigning a task to students is 
not enough for quality learning. Providing communication and collaboration is a critical part of online 
learning and thus the course design should consider offering opportunities for dialogue and feedback for 
students. 
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Should be “Incorporating current educational theory Educational theories can provide valuable 
recommendations and justifications for the instructional design activities and thus can help educators to set 
and evaluate appropriate tasks and activities for students. 

Should be “Deciding on the message before choosing the medium”: A common mistake nowadays is that 
educators often emphasize on the technology issues too much while neglecting some critical course design 
issues. Therefore, it is important to design the course related activities first and then consider what 
technology may better assist those activities. 

Some of the above mentioned instructional course design issues are not entirely new since they also exist in 
the traditional course design process. However, as Littlejohn emphasized, these issues seem especially prevalent in 
current online course design processes. For instance, because of the lack of technological skills in the online course 
delivery environment, the faculty often relies on staff members to design the Web pages and to develop the major 
portion of the online courses. As a result the technological issues are usually well addressed while the actual 
pedagogical issues are often neglected. This point is also well illustrated by Hara & Kling (1999); “There has been 
an unrelenting cycle of technology promotion and adoption in classrooms since the 1920s, where technology was 
introduced by enthusiastic advocates, such as administrators and researchers, only to fail because teacher lacked 
equipment, time and training.” 

It has becomes common knowledge that World Wide Web can be a powerful medium for teaching and 
learning at a distance. According to Khan (2001), “Numerous factors help to create a meaningful learning 
environment, and many of these factors are systemically interrelated and interdependent.” He developed a Web- 
based Learning Framework that categorizes these factors into eight dimensions: pedagogical, technological, 
interface design, evaluation, management, resource support, and institutional. 

While Khan’s holistic Framework provides the scaffolding for designing quality Web-based instruction, it 
seems too broad to provide practical guidelines on design details. There is no doubt that such a comprehensive 
theoretical framework is absolutely useful as a reference for designing an effective online course; however, it is not 
sufficient. Other practical considerations such as when faculty should start the preparation, what process they should 
follow, what external aids they may need, and what impact a certain design may have on students etc. are also 
critical for designing and developing an online course. University of Illinois Faculty Seminar Report (1999) 
summarized two sets of guidelines for designing and developing online coursework. The questions asked in the first 
set of the guidelines, practical considerations for faculty, can provide additional insights to capture the critical 
issues in the online course design from the faculty perspective. They are: 

Whom do I teach? 

It is critical to do learner analysis before building an online course. Not everyone is suitable to take online 
courses and not all programs can be delivered through the Internet. 

How do I teach? 

Choosing the right methods and media in relation to the nature of the course is critical. For instance, text- 
based CMC seems more appropriate for seminars while interactive and graphically based materials are 
better suited for courses that are traditionally taught in the lecture mode etc. 

How many do I teach? 

Keeping student-to-teacher ratio low is important to maintain a highly engaged online class. The 
importance of motivation and the learning community is also addressed in this section. 

How do I ensure high quality of online teaching? 

“Online course quality is best assured when ownership of developed materials remains in the hands of 
faculty members” rather than in the control of non-teacher course designers or administrators. Evaluation of 
learning effectiveness is also a critical aspect to ensure the course quality. 

Unfortunately, there are not many empirical studies that showed how educators were building online 
courses or how they were adapting the residential courses to the online environment. One possible reason may be 
due to the fact that every course is unique in terms of the course objectives, content, students, and resources. The 
uniqueness of each case may result in different instructional design processes and thus hinder the transferability of a 
case to other situations. However, as Dick & Reiser (1989) pointed out, instruction (sequences) generally have seven 
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common elements: “motivating the learner, specifying what is to be learned, prompting the learner to recall and 
apply previous knowledge, providing new information, offering guidance and feedback, testing comprehension, and 
supplying enrichment or remediation”. Since these common considerations exist in all types of instruction, educators 
can still learn a lot by just reading others’ experiences. A close examination of a practical case can provide valuable 
experience to the readers who may learn and adapt certain design strategies in to their own situations. 

Case selection and Course description 

The selected case is a computer mediated learning course (R547). The researcher is one of the teaching 
assistants (TAs) of this Web-based distance course and had experience assisting with the residential version of the 
same course in the past. R547 is a three -credit master’s level course on design, development and formative 
evaluation of computer-mediated learning programs. Students use software development tools to create and evaluate 
interactive lessons (detailed information can be found at: http : //www . Indiana .edu/~r547/) 

It is an intermediate level design and development course and thus the students enrolled in this class are 
fairly familiar with software and hardware applications from their previous class experiences. The faculty members 
who teach R547 have strong technological backgrounds. The traditional version of R547 has been taught for many 
years and it is still offered once a year to the residential students. It is important to note that the primary instructor 
of the online R547 did not have previous experience with teaching R547 although she took this course as a student 
several years ago. 

Fourteen students enrolled in the current online R547. All of them have full time jobs and most of them 
have families. Four students work for higher educational institutions, three students work in corporate settings, and 
seven students are K-12 teachers. 



Methodology 

Data collection methods 

According to Yin (1994), case study is an appropriate strategy when the nature of the research questions 
aims to discover “how” and “why” of certain phenomenon within real life context. The current study posed how 
and why the instructors and course designers are planning, designing, developing, implementing, managing, and 
evaluating their online course. It also provides how students feel about the major features of this Web-based course. 
The specific methods of collecting data are listed below: 

Instructor interviews A total number of seven people were interviewed including the primary instructors 
of both online and traditional versions of the R547, teaching assistants, technical coordinator, and institutional 
supporter. A 30 - 80 minutes, one-on-one interview was conducted with each of the participants. Each interview 
was audiotaped and partially transcribed. The interview questions were designed to capture the essential activities 
and decision-making points of course design and development process and also addressed other related questions. 
The interview questions can be categorized into a few major categories (for detailed questions, please see Appendix 
I, II, and III): 

Related background information of participants: This piece of information helps understand what kinds of 
knowledge and skills are involved in this online course design and development. 

What: What are the objectives, scope, resources, content and structures of the course? What activities and 
events did they conduct to accomplish what goals? Are they still making certain changes and adjustments to the 
course? What external help did they have? 

How: How did they start the design process? How did they decide on the delivery methods and media? 
How will student performance and achievement be measured? 

When: When did they start to prepare for the online course? How long did each critical step take? Was 
more time needed? 

Why: To find out the rationales behind major decision-making points that took place during the online 
design and development process. How did the online course differ from the traditional version of the course? Why 
were certain changes made? 

Document analysis Course syllabus, schedule, weekly readings and assignments, messages posted in the 
discussion forums, completed projects and reports are reviewed and analyzed various stages to provide additional 
information for a holistic understanding of the case. 

Participant observation As one of the teaching assistants in both the online and traditional versions of this 
course, the researcher herself participated and observed the detailed design, development, and evaluation process of 
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the instruction. Hopefully, such an experience can deepen the understanding of each phase of the development in 
great detail. 

Student survey on course evaluation A mid-term course evaluation of 44 items was sent to the students 
after the first 8 weeks’ class. This anonymous survey was active for three weeks and received 100 percent return 
rate. Among the 44 survey questions, the first 37 questions were 5-Likert Scale items (1= strongly disagree and 5= 
strongly agree) and the rest are open-ended questions. 1 am going to use some of the survey data to analyze what 
impact certain instructional design strategies have on students. 

Data analysis methods 

There are numerous instructional design models in the field of education. But almost all of the ISD models 
are based on the traditional generic ADDIE model (Kruse, 2002; Molenda, 2003). Although the ADDIE model has 
been criticized as too linear, inflexible, and time consuming, it is a systematic approach of designing instruction and 
thus provides a holistic perspective on the design and development process. Eurthermore, the preparation stage of 
most Web-based courses, if not all, takes a fairly long time, at least a couple months before the courses are taught. 

The word ADDIE represents each phase of the instructional design process: Analysis, Design, 

Development, Implementation, and Evaluation. Villalba and Romiszowski (2001) pointed out another category. 
Management. Issues of management seem especially critical in an online course environment due to its heavy 
technology-dependent nature. A good management strategy can ensure maintenance of high quality program 
throughout the entire teaching and learning process. Therefore, the course design and development actions are 
classified into the above six categories of an elaborated ADDIE model. A comprehensive summary of major 
changes of the course adaptation is provided in the Results section. The major changes and timetables went through 
the member checking process. 

The relevant student survey data were identified from the whole data set first. Next, the mean scores of the 
5-Likert scale items were calculated for each question. And then the findings were matched to the relevant design 
strategies to reflect the student reactions to major course features. A summary of the reactions and the implications 
is provided in a table. 

Results 

The collected data were analyzed in the following six categories of the elaborated ADDIE model: analysis, design, 
development, implementation, evaluation, and management. 

Analysis 

Early preparation: Eour months before the online R547 was offered, the instructor of the course asked to 
be a volunteer TA of the residential version of R547 to gain necessary teaching experience and also to get more 
familiar with the course content. At the same time she started to look for appropriate readings for her distance 
students. She thought that representative and high quality required readings are necessary for distance students since 
they will not have face-to-face lectures from the professor as the residential students have. 

Student survey of technology access'. About three months before the online class started, the department 
sent out a survey to verify what computer platforms (PC or MAC) and network connections the registered students 
had. Although not everyone was registered for the class at that time, they had a representative sample of the entire 
class. (2/3 of the class was registered at that time). Based on this student survey, the instructor contacted the 
University Information Technology Service (UlTS) to find out what new technologies can be employed to enhance 
the communication of the class. She had five different meetings with the director and staff of UlTS to discuss two 
issues. One is how to conduct Web conferencing with students in a real time (synchronous communication), and the 
other one is how to make class tutorial using screen and voice capture. 

Finding available support The primary instructor of R547 sent out an announcement to find those who are 
interested in being volunteer teaching assistants (TAs) in the department. Six advanced graduate students were 
interviewed and accepted as teaching assistants. The department also asked another advanced doctoral student to be 
the technical coordinator of this course. Thus a team of eight people formed to teach this online cohort of 14 
students. Based on everyone’ preference and skill set, the following roles were given to six volunteer TAs: 

Four frontline TAs: each will be in contact with 3-4 students on a regular basis to provide immediate 
feedback and help 

One pedagogical assistant and editor: since most of the TAs are international students, an English editor is 
needed. 

Technological TA: part of the course content requires students to have certain level of programming skills 
and a TA who knows programming is needed. 
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Student background information analysis: Students were asked to complete a pre-assessment survey on the 
first day of the class. This survey is designed to determine student background information such as where they are in 
their program, how many other courses they are taking at the same time, where they work and what their career 
goals are etc. During the first week of the class, the instructor conducted an hour long one-on-one voice chat with 
each student using either Netmeeting or Yahoo Messenger. The purpose of this synchronous communication was to 
know each of her students better and also to inform students that the class had officially began. Based on the pre- 
assessment survey and the one-on-one voice chat, the instructor divided the class into four different peer support 
groups. A corporate group, a higher education group, and two K-12 groups were formed. Each group has a primary 
TA assigned to them from the volunteer TAs. 

Design and development 

The design and development process of this course went hand in hand and it was almost impossible to 
separate each development activity from its design stage. Therefore, they are described together in this section. 
About two months before the class started, the instructor and the technical facilitator started the course design and 
development process. All the resources that were used in the residential R547 were copied from SiteScape Forum to 
the Oncourse course management system. Some of the materials were converted into Web pages in order to have a 
link provided. The residential version of the R547 syllabus and teaching strategies were modified in a few ways to 
accommodate the online nature. 

Changed the team-based project to an individual project: This modification is made to eliminate the 
possible frustration of teamwork at a distance. Students may save lots of time if there are less management issues. 
Furthermore, choosing a project in their primary interest may result in more engagement and motivation. 

Discussion forum: A weekly discussion topic is led by TAs based on each week’ readings. Each discussion 
topic is pre-designed and developed by 1-2 TAs first. Furthermore, the instructor and the pedagogical TA will 
further revise and polish the discussion topics before they are available for students. All students are required to 
participate in the weekly discussion forum since it is designed to partially substitute for the face-to-face version of 
lectures. 

Syllabus: Hardware and software requirements were added to the syllabus. Assignment and activities were 
modified to reflect the weekly schedule of the online course. Assessment criteria are somewhat different from the 
residential version. Since the project is changed from team-based to individual, there won’t be a peer evaluation 
component. Instead the discussion forum participation will be considered in the final achievement evaluation. 

Tutorial design: As students are required to build an online resume and a Web-based instructional project, 
they need to have individual server space where they can upload their Homepage and Web product. About six 
weeks before the class started the technical facilitator applied for server accounts for students, and created a step-by- 
step online job aid for the student’s use. The programming language portion of the course in the residential version 
needs two weeks (four sessions) to deliver face-to-face. The programming TA worked with the University 
Information Technology Center to develop a lecture CD in which the demonstration on computer screen and the 
voice are captured. Each student received a copy of the CD. 

Weekly instructor and TA meeting: Five weeks prior to the beginning of class, the first TA meeting 
(primary instructor and TAs) was held to determine the action agenda for designing and developing overall content 
focus as well as the content of weekly sessions. 

• What the general course structure would be 

• What media might be used 

• What the online discussion topics should cover 

• What the discussion topic order should be 

• Who should lead which topics 

• What the class schedules (reading & assignment due date) should be 

• What the evaluation criteria would be 

Discussion topic design: Each TA is responsible for leading 1-2 discussion topics throughout the semester. 
Who should facilitate what topic was determined based on TAs’ preferences, background knowledge, research 
directions and proficiency level. TAs are asked to send a draft of topic outline two weeks prior to the discussion 
week. The primary instructor and the pedagogical TA are responsible for checking whether the designed activities in 
each topic are appropriate and engaging. Since the online discussion topics are a part of the substitute for the face- 
to-face lecture, it is critical to prepare each topic carefully. This part of the content design and development 
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continued through two-third of the semester. Therefore, the design and development of course content were not 
completely ready before the semester started. 

Another part of the substitutes for face-to-face lecture is synchronous live lectures that are given by four 
guest speakers during the second half of the semester. The primary instructor believes that synchronous discussion 
and live lecture can be a valuable learning experience for distance students once the technical issues are overcome. 

Implementation 

The implementation phase was started once the first course segment, the modified syllabus, was uploaded 
and tested on the course management system. The technical coordinator needed to convert various documents such 
as the syllabus, schedules, discussion topic outlines, and some other course materials into HTML Web pages. 

During the couple months before the class started, the technical coordinator spent about twenty hours a week on 
converting documents into Web pages, testing their usability, copying/moving certain course materials from the 
residential course site and setting up various work folders for students and TAs on the course management system. 
Once the class started, the technical coordinator mainly needed to maintain the course management system and it did 
not take too much time. 

After the class started, TAs were asked to contact each of their students for a one-on-one or as a small 
group voice chat. This was to help student prepare for the synchronous meeting and lectures in the near future. If 
they learnt to use the Web conferencing tools beforehand, it could make their learning easier later in the semester. 

Evaluation 

During the content development, various documents went through recursive evaluation process. 

The technical coordinator would first develop the Web pages and the primary instructor would evaluate the quality 
of the work. If they were not satisfactory, further revision would be needed. According the technical coordinator, 
the re-designing and re-doing took much more time than she had expected. 

Each online discussion forum outline was developed by TAs who were responsible for that topic. If the 
primary instructor and pedagogical TA felt it needed revision and re-design, they would have a small group meeting 
with the initial designers to revise the topic outline until it reached a satisfactory stage. 

A mid-term course evaluation survey was sent out to each student for determining the potential 
enhancements for the overall course design, development, and evaluation. Plenty of positive feedback was received 
from students. 

Management 

In order to maintain a well working online course, the following strategies are employed; 

• Weekly announcement on Oncourse to inform student what the focus of the week is and what the 
deliverables are 

• Weekly TA meeting to discuss what is happening with each group and what actions should be taken 
next etc. 

• TAs are required to check their Oncourse email once a day 

• TAs are asked to respond to student questions in 24 hours 

• TAs are required to give out prompt and extensive feedback on student work 

• TAs also need to contact student at least once a week either one-on-one or as a small group 

Major changes made during course adaptation: 



Residential R547 


Online R547 


Justifications 


Team-based project 


Individual project 


To eliminate teamwork frustration at distance 
To increase engagement and motivation 


Lecture (twice a week) 


Facilitated discussion forum 
(weekly) 


Using structured weekly discussions to partially 
substitute for the lectures. The purpose was to 
promote deeper thinking and engagement about 
key concepts that related to their development 
work. 


Less readings (no required 
textbooks; supplemental 
online sources provided) 


More readings 

(Structured weekly readings based 
on two required textbooks and 
supplemental online sources) 


This was designed to provide more structure for 
students, as a substitute for absence of face-to-face 
lectures. 
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Hands on demonstrations 


Job aids and CD tutorials 


Since the hands on demonstration were not 
possible at distance, online job aids and CDs were 
produced. 


Use of course management 
system: 

Reference resources 
Team working space 
Gradebook 


Use of course management 
system: 

Reference resources 
Team working space 
Gradebook 

Email communications 
Class discussion forums 
Announcements 
Calendar with task items 
Online surveys 


The use of various communication modes and 
weekly announcement of assignments and 
deliverables seemed critical in online R547 since 
the instructor can not share information and 
remind students in person 


Face to face consultation 
and problem solving 


Synchronous (voice/text chat) and 
asynchronous (email/discussion 
board) communications 


Based on the group members’ computer platforms 
and broadband connections, different software was 
used to exchange ideas and to solve problems. 
Students were required to learn a simple voice chat 
software (i.e. Yahoo Messenger, Ivisit), a web- 
conferencing software (Groove, Netmeeting), and 
course management systems (Oncourse and SSF). 
The underlying assumption was that the context of 
different problems and users required different 
methods of communication. Voice chat was used 
as a redundant, backup system in case firewalls 
and/or connection speeds prevented the use of web 
conferencing software. 


No assigned TAs for each 
student 


Each student has a primary TA to 
contact 


To provide quality technical support with one-on- 
one tutorial support 

To provide rapid turnaround of the feedback 
To eliminate student feeling of isolation 


Student to TA ratio is 8:1 


Student to TA ratio is 3:1 


It was believed that lower student to TA ratio can 
help provide quality and prompt feedback for 
students 


No regular instructor & TA 
meeting 


Weekly TA meeting 


To keep track of what is happening 
To provide consistent feedback to students 
To implement consistent action plans 


No assigned peer support 
groups 


Assigned peer support groups of 3- 
4 students based on their work 
background 


To reduce feeling of isolation 

To provide peer perspectives and support 

It was believed that people with similar experience 

tended to provide more relevant feedback to each 

other 


Peer evaluation is 10% of 
total grade 


Class forum participation is 10% 
of total grade; Written peer support 
group feedback is 10% of total 
grade 


Since the project work is individual-based, there is 
no need for peer evaluation on project 
development process. Instead, the discussion 
forum participation and peer communications are 
more critical for online courses 



Students’ reactions to the course features 

This section of the report is based on the mid-term course evaluation. Overall, the feedback was very 
positive. Important findings that are associated with the changes made during course adaptation are analyzed below: 
Reactions to individual based projects: In the course evaluation, thirteen out of fourteen students think 
having the class projects as individual-based projects enhances their learning experience (M= 4.29, out of 5-Likert 
scale). One student wrote, “It has been very motivating to me and I recommend that you continue to have the 
flexibility to allow students to do projects that are meaningful to them.” Another student expressed a similar opinion 
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“I appreciate that I’m designing things that I can actually use.” On the other hand, another student wrote, “I’m 
intimidated by the prospect of putting this all together myself on Dreamweaver. . . I need a technical advisor. . .” This 
made us question whether this modification was a total success. In general the team-based projects are to promote 
collaborative learning. It is believed that team members’ complimentary skills are necessary to accomplish the task 
successfully and that they can learn from each other by working together. When the course projects were assigned 
as individual tasks, instructors need to consider what other relevant changes should be made to align with that 
modification? Is it fair to assign the same workload for one student while three to four students used to work on one 
project in the face-to-face version of this course? While we try to keep a high quality online course in which we 
make distance students learn no less than what residential students learn on site, we are lack of experience how to 
balance student workload with various changed strategies we made in distance education. 

Discussion forum: Eleven out of fourteen students agree the online forum discussions promote their 
understanding of the concepts in this course (M= 3.7). In order to enhance the participation rate and engagement, 
the weekly discussions had various role assignments for each student either in their peer support group or in the 
entire class. However, students seem less happy with such structured discussions (M= 3.07). The open-ended 
questions revealed students’ unenthusiastic attitudes toward the structured weekly discussions such as “Don’t ask 
people to be wrappers - it is way too much work. Just allow us to freely discuss a topic”, “It would have been useful 
for the instructor to provide some type of summary of the topic - to capture key thoughts”, and “Some of the weekly 
discussions are a little too much with the other work we have.” These comments emphasized the importance of 
pedagogical aspects of designing online discussions. While it is critical to have such a discussion forum for distance 
students as a partial substitute for the live lectures, it should be carefully designed and cautiously implemented in the 
real settings. Balancing workload and providing key takeaways of the discussion topics may enhance the 
effectiveness and student satisfaction of discussion forum usage. 

Communication mode: Upon being asked to rate the overall value of each synchronous and asynchronous 
communication tools, students voted email of great value over other media. 

E-mail: M=4.71 

Asynchronous discussion forum: M=4.43 

Synchronous voice chat: M=3.93 

Synchronous text chat: M=3.86 

Enhanced voice chat with co-browsing and co-viewing of documents M=3.57 

These data showed that students prefer to communicate asynchronously than to communicate 
synchronously. Students were introduced to several different synchronous and asynchronous software applications 
such as Yahoo Messenger, Ivisit, Groove, Netmeeting, Oncourse, and SSE. The purpose was to accommodate 
students with different network connections (broadband vs. dial up) and computer platforms and to provide flexible 
communications. However, students expressed frustrations over learning to use so many different applications: 

“We seem to be experimenting with a lot of different tools and modes. It feels a bit disintegrated to use Oncourse, 
SSE, Groove, Netmeeting, etc. Eor future reference I would narrow it down to fewer communication tools, fewer 
new software products to learn” 

“Too many options make for some confusion. Eind something that works and stick with it. . .” 

“At times, we seem to want to use online technology when a simple telephone call would do.” 

“The interactive tools proved to be a frustrating experience. We all seemed to have some problems 
with them. In theory, tools such as Groove and Netmeeting are wonderful. In application, it may 
be a different matter. . .” 

One suggestion of these survey data is to minimize the types of software applications for student use. While 
synchronous and asynchronous communications both have their own value, it is important to balance their 
proportions in distance education. 

Reactions to TAs and the instructor: All students agree that TAs have provided quality, supportive feedback to them 
(M= 4.57). Student gave a very high remark to the voluntary TAs. The open-ended feedbacks of “I am totally 
impressed with the dedication of the TA’s and the instructor to helping the class” and “It was nice to see the T.A. 
paying attention to the support group and checking in with folks” help illustrate students’ positive attitude toward 
TAs. However, students may not want TAs to be involved in every aspects of the course. One student wrote as: 
“Although synchronous chats for the entire class tend to be inefficient and chaotic, because they 
end up being multithreaded, I think they are helpful to provide an ongoing focus to the class. 

These should be conducted by the instructor, with little or no participation by TAs”. 
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Although students appreciate the help from TAs, they tend to prefer direct support from the instructor at 
certain times, if not always. These empirical data poked a few questions for future research; what should the TA’s 
role be in distance education and how much should a TA be involved? Another relevant question is what the best 
(cost-effective) student to TA ratio is? 

Peer support: Based on the students’ work sector, preference, and relevant experience, the instructor assigned each 
student to a peer support group of 3-4 people at the beginning of the semester. The purpose was to eliminate the 
student’s feelings of isolation and to provide peer perspectives and support. All students admit that peers in their 
peer support group have been supportive (M= 4.5). “I’ve really enjoyed the group and class discussion forums. It’s 
helpful to get input from my cohort peers’’, “working with my peer group on projects [is the favorite aspect of the 
course]”, “The peer support groups have been more valuable than the larger group discussions. I appreciate the 
perspective of the corporate and higher education groups, but my goals focus primarily on the K-12 arena” and “my 
least favorite are the discussions with the entire class. I don’t feel the feedback and time spent is helping me like the 
smaller group discussions have.” However, not all students like to be assigned into such a background-based peer 
support group. One student wrote, 

“I don’t like the segregation of K-12, higher ed, and corporate. While it is nice to have a peer 
group occasionally, I really miss the interaction with others and the valuable ideas and help I gain 
from the class as a whole. This format seems to be very limiting and I know I haven’t learned as 
much as I would have if I interacted with others in the class as well as those in my peer group.” 

There is no doubt that peers can provide valuable support and insight for each other. But the critical issues 
seem two fold; what are the best ways to assign peer support groups and how to balance small group and the entire 
class interactions. The individual differences seem to be at play as well. There may not be an easy solution to 
satisfy everyone’s need and expectation on this issue, but future studies should be dedicated to finding better ways to 
accommodate the students’ preferences. 
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Appendix I 



Student survey data analysis 



Major modification 


Student feedback 


Implications 


Individual-based project 


Advantages: 

-Provides flexibility over project selection 
-Promote motivation and engagement 
-Eliminate the teamwork frustration at distance 
Disadvantages: 

-Increase student workload 

-Increase frustration on some students who do not have all the skills required for finishing the 
project successfully 


Need to consider what other 
relevant changes should be made 
to align with this modification 


Structured (role assigned) 
online discussion 


Advantages: 

-Promote deeper thinking and engagement about key concepts 
Disadvantages: 

-Inflexibility 

-Too much work while there are other assignments due at the same week 
-Student easily get lost and do not know what the key points are 


Provide key takeaways of the 
discussion topic and balance 
workload 


Synchronous and 

asynchronous 

communications 


Student preference: 

-Email 

-Asynchronous discussion forum 
-Synchronous voice chat 
-Synchronous text chat 

-Enhanced voice chat with co-browsing and co-viewing of documents 
Students frustrated to learn too many communication tools including Yahoo Messenger, Ivisit, 
Groove, Netmeeting, Oncourse, and SSE 


Minimize the number of 
communication software 

Balance the proportion of 
synchronous and asynchronous 
communication 


Low student to TA ratio 


Advantages: 

-Provide prompt and helpful feedback on student work and questions 
-Eliminate the feeling of isolation 
Student concern: 

-TAs are doing some jobs that should be done by the primary instructor 


What should TA’s role be in 
distance education? 

What is the best (cost-effective) 
student to TA ration? 


Assigned peer support 
groups based on students’ 
work sector, preference, and 
work experience 


Advantages: 

-Provide most relevant feedback 
-Promote deeper discussion on the focused topic 
Disadvantages: 

-Decrease the chance of getting feedback from others who are from different background and 
thus limit the possibility to provide diverse perspectives 


What are the best way to assign 
peer support groups? 

How to balance small group and 
the entire class interaction? 
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Conclusion 

In this case study of examining instructional design and development of a Web-based course, the interview 
data were analyzed using the elaborated ADDIE model. The major changes made during course adaptation from 
traditional version to online version were highlighted. And the consequences of major changes on students were 
analyzed from the student feedback received during the mid-term course evaluation. Useful implications and 
possible research questions for future research studies were drawn from the analysis done in the results section. 

The major limitation of this study resides in the fact that this is a single case study with 14 students. It is 
also important to note that this course is offered by Instructional Technology department. This means the instructor 
and TAs are experts in instructional design and development. Therefore, the current case is an atypical in this sense. 
Since each course is unique in many ways, the result of this study should be referenced cautiously upon 
implementation in different situations. Similar case studies should be conducted in the future to provide more 
powerful results for educator use. 
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Informing Instructional Systems: 

The Concept of Intention in the Context of Learning and Instructional 

Theories 
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Abstract 

This presentation paper discusses the concept of intention in the context of learning and instructional 
theories that inform the field of instructional systems and human performance, e.g. behaviorism, information 
processing, and motivation. The closing section speculates benefits to the field of instructional design from 
incorporating intention research. The conclusion advocates the use of the intention conception with the Instructional 
Systems applied research activities to build a model for complex affective skill acquisition. 

Introduction 

Ruth Kanfer of the University of Wisconsin has provided statements regarding current research efforts, and 
asserts that they all relate to motivation: assessment of intrinsic outcomes, information processing, changes in 
motivational orientation, and developing cross-disciplinary models of motivation. “Contemporary findings indicate 
that differential activation of particular motive states depends on the individuals’ previous history and environmental 
factors. . ..Unlike earlier theory, the emphasis in current research is one of the factors that influence the way 
information is processed, that is, changes in motivational orientation rather than on the influence of specific motives 
per se. Focusing on these facilitates opportunities for developing more comprehensive, cross-disciplinary models of 
motivation” (Kanfer, 1990). 

In essence, the described line of research is of professional interest and personally valued as relevant to the 
instructional systems research and practitioner communities. The expressed research interests encompass affect and 
how it moderates (positive and negative) information processing in learning, behavior, and performance. The 
interest in intention research is an attempt to understand human learning, behavior, and performance, explain the 
variance, and then recommend how to effect change. Related research interests include: affect, valuing, cognition, 
volition/cognitive choice, decision-making, and metacognition. From cursory readings it was observed that the 
intention construct is related to all of the referenced research areas and supports the processes of goal setting and 
self-regulation. “The constructs of motivation subsume the determinants and processes underlying the development 
of intentions, choice behaviors, and volitional activities. The products of these motivational processes are the 
individual’s overt and/or covert behaviors” (Kanfer, 1990). A second observation is that the majority of the research 
interests share conceptual origins with research in motivation. 

The Intention Construct and Theory of Planned Behavior 

According to Icek Ajzen and Martin Fishbein, intention captures the motivational factors that impact 
behavior. They are indications of how hard people are willing to try, and how much of an effort they are planning to 
exert, in order to perform that behavior. “These intentions remain as behavioral dispositions until at the appropriate 
time and opportunity an attempt is made to translate the intention into action” (Ajzen, 1988). 

Intention predicts behavior. 

Available evidence supports the idea that intentions are close antecedents of overt actions. The predictive 
validity of intention is usually significantly greater than that of attitudes toward the behavior (Manstead, 1983). 
Intention has been found to predict a variety of corresponding action tendencies, ranging from simple strategy 
choices to “actions of appreciable personal or social significance, such as having an abortion, smoking marijuana, 
and choosing among candidates in an election” (Ajzen, 1980; Ajzen & Fishbein 1970). Although intentions can be 
predicted with a high degree of accuracy, they will decline with the amount of time that intervenes between 
measurement of intention and observation of behavior (Songer-Nocks, 1976; Fishbein & Coombs, 1974; Sejwacz et 
ak, 1980). 

According to Ajzen’ s theory of reasoned action, intentions are a function of two basic determinants, one 
personal in nature and the other reflecting social influence (the person’s attitude toward the behavior). This attitude 
is the individual’s positive or negative evaluation of performing the particular behavior of interest. The second 
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determinant of intention is the person’s perception of social pressure to perform or not to perform the behavior under 
consideration (subjective norm). 

The theory of planned behavior is considered an extension of the theory of reasoned action. The theory of 
planned behavior (TPB) adds a third determinant of intention - perceived behavioral control (Ajzen, 1985; Ajzen & 
Madden, 1986). The TPB incorporates perceived behavioral control as a factor inhibiting or encouraging one’s 
intentions towards behavior. Study data indicates that perception of behavioral control can have an important impact 
on a person’s behavioral motivation (Ajzen, 1988; Ajzen and Madden, 1986; Schifter & Ajzen, 1985). 

Generally speaking, people intend to perform a behavior if their personal evaluations of it are favorable, if 
they think that important others would approve of it, and if they believe that the requisite resources and opportunities 
will be available. Subjective evaluation of those outcomes, reveal why a person holds a favorable or unfavorable 
attitude toward performing the behavior. Taken together, this informational base provides us with a detailed 
explanation of a person’s tendency to perform, or not to perform, a particular behavior (Ajzen, 1988). 

Relevancy of intention research to learning and instructional systems theories 

“Expectancy-value theories, intrinsic motivation theories, self-regulation theories, and attributional theories 
all have stressed the energizing role of affect and emotion as they influence the individual’s organization and 
interpretation of knowledge and search for information ’’ (Kanfer, 1990). 

It is speculated that training situations which require affective and behavioral change to obtain performance 
will benefit the most directly and immediately from intention research results. Additional plausible benefits from 
intention research results within learning and instructional systems are: 

• development of learning conditions for affective domain skills, 

• design, development, and implementation of methods, strategies, and tools for affective domain training, 

• affect related information processing adjustment to other cognitive-behavioral change interventions, and 

• inform current performance improvement strategies of additional techniques. 

The intention research is compatible with other motivation theories, such as Keller’s ARCS motivation 
theory, McClelland’s need for achievement theory, Atkinson’s anxiety and learning, and learning theories such as 
Behaviorism and social learning theory (Driscoll, 2000). Keller’s ARCS model (1987) has provided principles to 
gain attention and build relevance, confidence, and satisfaction within instructional materials. The model is 
effective to move an individual through the instructional material while maintaining interest. It is believed that 
intention research results will support the principles of motivational design (of instructional materials) by providing 
specific ratio, emphasis, order, or repetition required for predominantly affective skill acquisition. Also, the intention 
research tools could be supportive in diagnosing why a learner does not engage with motivationally designed 
instructional materials. Consider this analogy, “You can bring a horse to water, but how do you get him to drink it?” 
How does one engage the horse in voluntary drinking of water without the internal drive of thirst to stimulate the 
desire for water? Likewise it is assumed that the water is well-designed instruction with plenty of motivational 
characteristics to maintain interest after drinking begins. The analogy is used only to illustrate perceived current 
limits of learning and instructional theories. 

Behaviorist strategies like Hull’s drives toward anticipated goals (Driscoll, 2000) are effective after 
behavior is engaged (directly or indirectly valued) and rewarded. For example, a behavioral approach to increasing 
duration and frequency of study behavior must make a pay off with rewards that can compete with other rewards 
gained from competing behaviors (social, recreation, work, sleeping). The solution is simple - provide for multiple 
short-term payoffs (Wager, 1977). What is an effective intervention when a behavioral outcome is not valued, 
perceived as controllable, or the performance to reach that goal carries too many negative consequences? Can a goal 
be set for something that is perceived as irrelevant, having negative social consequences, and creating negative 
behavioral outcomes? The researcher’s objective is to use the intention construct to study this type of problem in 
human learning (behavior acquisition) and performance. It is believed that part of the answer is associated with 
intention antecedents (behavior evaluation, subjective norms, and perceived behavior control). 

It is plausible to describe the antecedents of intention (positive or negative evaluation of performing the 
particular behavior, and subjective norms) as further elaborations of Ebbinghaus and Thorndikes’ theories of 
association, and later behaviorists’ theories of counter-conditioning (systematic desensitization). Results from 
intention research can inform how to better organize learning events (instruction) to counter negative affect, 
negative associations, negative conditioning and to allow more positive information processing to occur without 
dissonance. For example, some populations of learners (young and old) continue to believe and state, “I can’t learn 
,” “I can’t do ,” “I could never learn/do .” It’s conceivable that an effective intervention would include 
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desensitization of prior negative associations and conditioned responses. Eventually, the goal is to create an 
algorithm to determine countering per antecedent or desensitization per antecedent. For instance, to advance an 
individual to behavior acquisition, how much emphasis should be placed on perceived behavioral control in relation 
to his negative evaluation of the behavioral outcome? 

McClelland’s theory of need for achievement does not seem to address voluntary behavior acquisition and 
performance in the realm of non-consequence and negative associations. However, Atkinson’s anxiety and 
performance theory may be complimentary to interventions related to the perception of behavioral control. 

Bandura’s social learning theory has similar tenets as the theory of planned behavior, e.g. Bandura’s self-efficacy as 
a belief system that is causally related to behavior and outcomes (Driscoll, 2000). It appears that perceived behavior 
control and self-efficacy are related to Bandura’s self-efficacy belief system. The theories of how it mediates 
performance are the most comparable of all the theories I have reviewed within learning and instructional systems. 
And finally, Locke, Latham and Zimmerman’s theories of goal setting and self-regulation (Driscoll, 2000) are 
critical but begin after behavior intention is present. It is my opinion that proximal goals of XY behavior outcome or 
performance would be the most immediately relevant to populations experiencing negative affect towards XY 
behavior or performance. 

A plausible use of intention research is to inform or explain variance of performance especially when the 
instructional materials were designed with sound design principles. As Kanfer states, cross-disciplinary and 
comprehensive models better explain human behavior and its variances. Perhaps a comprehensive motivation model 
would provide greater understanding of how to engage negative affect, build intrinsic value for a particular behavior, 
address variations of subjective norms, and increase perception of behavior control. 

Several societal, organizational, and personal problems have a solution available but it is dependent upon an 
individual’s choice to change behavior; the individual must choose to do “other.” Ajzen and Fishbein et ah, and 
Prochaska et ah, have demonstrated that choosing to “do other” is quite complicated and certainly requires more 
than an informational pamphlet, campaign, or workshop. Remember, many of the choices require the extinction of 
habit (or norms) without direct or immediate consequences (pain/reward) to support the performance (with the 
exception of select safety behaviors). 

Therefore, it is believed that intention research will 1) provide greater empirical basis to understand human 
learning, behavior, and performance, 2) inform training and tool design for difficult affective training situations, and 
3) inform strategies to effectively address subjective norms and perceived behavioral control. 

What is the relevancy of intention research to human performance technology? 

One plausible benefit is that Instructional Systems practitioners and graduates have an applicable model 
when the performance context is void of appropriate reward, control, or policy systems. During academic training 
graduates learn to identify more factors affecting performance, and learn an intervention to increase potential for 
performance among low performers. Both graduates and practitioners add another valid model with solid research 
foundation to their professional toolbox. Cross-disciplinary knowledge broadens placement possibilities into stable, 
growing, and well-funded sectors of American economy. In summary, it provides another tool to affect behavioral 
change and performance in the absence of controls and consequences (voluntary behavior). 

Adding knowledge to learning and instructional systems field 

“Another contemporary trend in motivation theory is the development of theories attempting to unify 
prevailing motivation constructs and cognitive, information processing psychology . Researchers attempting to forge 
an integration of cognition and motivation have focused on meta processes such as metacognition and 
metamotivation - that is, the cognitive and motivational processes that presumably govern the coordination of effort 
and skills . Developments during the past 15 years signal the beginning of a new phase in this area. One indication of 
change is the increased attention given to coordination of theoretical approaches whose origins trace to different 
subdisciplines of psychology. For example, it is increasingly common to see studies that build on an integration of 
concepts from clinical self-regulation approach, behavior modification approaches, goal setting and cybernetic 
control theories ” (Kanfer, 1990). 

Is it possible to change affect through training design? Is it possible to increase learning efficiency, 
increase behavior change, and improve performance by addressing an individual’s: 

• positive or negative evaluation of performing the behavior(s), 

• subjective norms, and 

• perception of behavior control? 
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Some researchers have said, “Yes, it is possible to change affect.” (Ajzen, 1988; Ajzen, 1985; Manstead et 
ah, 1983; Prochaska et ah, 1994; Prochaska et ah, 1994b; Prochaska, 1994c; Prochaska et ah, 1993). The theory of 
planned behavior (and the transtheoretical model) provides results worth considering when designing an effective 
intervention for highly affective behaviors. 

Design interventions (in the broadest sense) effect the adoption and extinction of specific behaviors born 
out of choice and managed by the individual. The following list is speculation of potential applications of a valid 
“intention” intervention: 

• When aptitude is present but performance is low (high aptitude & low performance). 

• Learners who present reluctance with learning tasks or whole skill sets: I can’t write, I can’t do math, I 
can’t spell, I don’t like to read. 

• Learners deficient in complex cognitive skills for their grade level/age (e.g. reading, math, writing, 
science). 

• Learning problems that include evaluation or “values” beyond learning verbal information, application of 
rules, and conceptual recognition. 

• Assistance in designing interventions, treatments, programs, training, lessons, computer-based instruction, 
management plans, and online tools. 

• Informed and more comprehensive theories, models, interventions, and techniques to be used by 
consultants, managers, performance technologists, change agents, preceptors, mentors, coaches, teachers, 
professors, parents, child care professionals and other change agents. 

The following are performance problems that engage affective and cognitive learning domains. The results 
of intention research could be applied to the following behavioral-performance areas: 

• Reading adoption (literacy, higher reading levels, read more frequently) 

• Illiteracy prevention (teachers, parents, and guardians) 

• Financial behavior (debt management and counseling) 

• Safety behavior (worksite, travel, and home) 

• Parenting skills & professional childcare providers 

• Healthy Baby Healthy Mom (prenatal and 0-6 programs. Chiles Center/DCF) 

• College enrollment programs for 1 st generation 

• College and high school drop out prevention 

• Food safety behavior 

• Driving safety behavior 

• Health risk behavior (prevention of: disease, obesity, drug addiction, drunk driving) 

• Cancer prevention (mammary, testicular, skin) 

• Safe sex behavior 

• Juvenile truancy behavior 

• Work absenteeism and school absenteeism 

• Work productivity and presenteeism 

• Performance evaluation and appraisal 

• Work relocation programs 

• Racial integration 

• Doctoral candidate drop out prevention 



Conclusion 

There have been sufficient studies indicating that complex skill acquisition requires sustained attention and 
practice, but what is still unclear is the requisite affect of the individual to engage in the task and adopt behaviors for 
successful performance. “Little empirical research exists on the relationship between instructional methods and 
activation of these different motives. Second, Lepper (1985) notes that little is known about what triggers different 
intrinsic motives during skill acquisition or even when motivational interventions should be implemented” (Kanfer 
1990). 

In summary, successful applied research in the affective domain could build a model for complex affective 
skill acquisition. 
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Using The T.E.A.C.H. Act To Enhance Online Courses: Guidelines and 

Strategies 
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Abstract 

The TEACH Act was signed into law on November 2, 2002. There were two significant changes to the law: 
1) rewriting Section 110(2) and adding a new Section 112(f). These sections now allow instructors in online 
courses to display and perform copyrighted works in greater amounts than before. This law recognizes the growing 
use and respect for online distance education, which will continue to grow at 20% per year by creating a new term 
in the law-“mediated instructional activities". Options for online course display and performance of copyrighted 
works have increased, though online uses of materials are less and more circumscribed than that of face-to-face 
instruction. Instructors must use an array of options to develop course materials: “fair use”, royalty-free and public 
domain materials, the TEACH Act provisions to display graphic and perform audiovisual works, the Educational 
Multimedia Fair Use Guidelines when the TEACH Act does not address a particular use, and permissions. 

Strategies and resources for using the T.E.A.C.H. Act to develop and use online course materials are provided, as 
well as a checklist of “do’s”, “don’ts”, and “tried and trues” to use in this process. 

What Is the TEACH Act? 

The T.E.A.C.H. Act is the Technology, Education And Copyright Harmonization Act of 2002 and was 
signed into law by President Bush on November 2nd, 2002. The TEACH Act amended the Copyright Act in two 
significant sections of the law; 1) by rewriting Section 1 10(2) and by adding a new Section 1 12(f). These sections 
now allow instructors of online courses to display and perform copyrighted works in greater amounts than before. 

Why Is It Important? 

This law recognizes the growing use and respect for online distance education, which is growing rapidly 
(Camavale, June 13, 2003). Eall, 2002 enrollment in higher education was 1.6 million students, and will continue to 
grow at a rate of 20% a year (Carnavale, June 13, 2003). A recent study of 1,600 institutions found that a majority 
of academic leaders (57 percent) believed that the learning outcomes in online education were “equal to or superior 
to those of face-to-face instruction” (Sloan Center for On Line Education at Olin and Babson Colleges and The 
Sloan Consortium, 2003). This Act codifies the recommendations requested by the 1999 U. S. Copyright Office’s 
Report. Moreover, research on online courses has shown that the average distance course is roughly the same size as 
a face-to-face course at a similar level, graduate and undergraduate (American Eederation of Teachers, 2001; 
McKenzie, Mims, Kirby, & Waugh, 2000; Wolcott, 1997). Lastly, encryption mechanisms, while not “foolproof’, 
have improved substantially (Talab, 2003). 

Can I Use Materials in The Same Way As in Face-To-Face Instruction? 

No. While “reasonable amounts” of materials may be used in online courses, it is still less than in face-to- 
face instruction. 

Make no mistake; While the TEACH Act is a major improvement over the previous version 
of Section 1 10(2), the law still imposes numerous requirements for distance education that 
reach far beyond the modest limits in the traditional classroom (Crewes, 2003). 

What Are My Options, Now, for Using Materials in Online Courses? 

The TEACH Act enlarges the amount and type of materials that can be used in online courses, in addition to 
“fair use” and the Educational Eair Use Guidelines for Multimedia (Harper, 2002). While permission is always an 
option, instructors can “shop” for the use that best suits their particular instructional needs. You have these options; 

1. Rely on “fair use”. Section 107 of the Copyright Law applies to the use of works in reasonable amounts for 
instructional purposes in much the same way that they are used in face-to-face instruction. This use is in 
addition to the TEACH Act (Gasaway, 2001). Works that arenot allowed to be copied are works for the 
digital distance education market, unlawfully made works, textbooks, and coursepaks. Check to see if a 
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use is covered by “fair use”. More information on “fair use” can be found at the Stanford University Fair 
Use Site: http://fairuse.stanford.edu/ 



2. Use royalty-free and public domain materials. These include materials from the Internet that are legally 
royalty free, public domain, most government, or other works that were created for free distribution. For 
example, there are several sites that have royalty-free graphics, such as Public Domain Pictures 

http ://camt. nvfa.org/artsites/ . The National Archives Research Room http://www.archives.gov/ and Kathy 
Schrock's Guide to Educator's http://school.discoverv.com/chpart. A site with an excellent list of royalty 
free graphics is at http://www.bsu.edu/librarv/thelibraries/units/copvright (Dolak, February 14, 2003). 

Most NASA images and all National Archives images are in the public domain. Doing a search on 
Google’s Image Search section ( http://www. google.com) will retrieve many images, most of which are 
copyrighted, so it is wise to check each site that contains an image. Limiting the search to “.gov” will 
retrieve images that are developed by the government. Most government images are in the public domain, 
unless a government contractor has taken the option of copyrighting the work. In which case this will be 
stated on the site. Otherwise, search on “public domain”; “royalty-free” or “free” graphics will yield 
results. 

3. Use the TEACH Act to display and performance of audiovisual works, such as film stills and clips 
(including opera), for music, etc. Laura Gasaway’s Comparison Chart for what may be performed and 
displayed in online courses is helpful in making decisions ( http://www.unc.edu/~unclng/TEACH.htm ). 

The American Library Association (ALA) has several sites that are also helpful. ALA often moves its 
sections around on the site. It is best to go to http://www.ala.org and search on the TEACH Act for Ken 
Crewes’ explanation of the TEACH Act and “Best practices” for using Blackboard. Other excellent 
resources on this Act are the “TEACH Act toolkit” at http://www.lib.ncsu.edu/scc/legislative/teachkit/ . 

And the Center for Intellectual Property and Copyright in the Digital Environment at 
http://www.umuc.edu/distance/odelFcip. 

4. When the length or use needed is not covered by the TEACH Act then turn to the Educational Multimedia 
Lair Use Guidelines. These Guidelines are not the law but were agreed upon by a large majority of 
educational media producers. In effect, these producers agreed not to sue anyone who uses amounts of 
materials stipulated in the Guidelines. This is a “safe haven” because they give more “leeway” in film 
clips, music, etc., and allow projects with these clips to be shown on protected networks. This is an 
important consideration when using PowerPoint and other programs for student projects. A facsimile of 
these Guidelines can be found at http://www.libraries.psu.edu/mtss/fairuse/guidelines.html and an 
explanation can be found at the University of Texas site at 

www.utsvstem.edu/ogc/intellectualproperty/ccmcguid.htm/ . Three short charts on using these guidelines 
with students can be found in Commonsense copyright: A guide for educators and librarians (1999). One 
is re-printed below (p. 213): 

Table USimplified Fair Use Guidelines for Educational Multimedia Student Form - Example One 



Database/Table 


Motion Media 


Music 


Images 


Text 


10% or 2,500 


10% or 3 minutes 


10% or 30 seconds 


5 from 1 


10% or 


fields or cell entries 




from 1 work or 


artist or 15 


1000 






extracts from 


images from 


words 






several works 


a collective work 





Credits: 

1 . When you use someone else’s work, give credit. 

2. Place the copyright information on a “Credits” screen at the end. 

3. Make a screen to show at the beginning that says “Pair use amounts used. Purther use is restricted.” 

4. If you follow these rules then you may enter your project in a local, state, or national contest. 
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How Can I Use These Options To Benefit My Online Courses? 

The good news is that these options may be used together or separately. When approached from this 
standpoint then use is widened. 

Basic Changes in The TEACH Act 

This law made five fundamental changes that assist distance education (Gasaway, 2001; Lutzker, 2002). 

1 . The categories of works that can be performed now include limited and reasonable portions of works that 
used to require permission and/or licensing. However, this does not include using clips from works 
produced primarily for educational purposes. As an example, if you want to use a video from The 
Association for Supervision and Curriculum Development, which specializes in exclusive productions for 
educational purposes, you will still need their permission to transmit their material, which is produced for 
educational purposes. 

2. That physical, face-to-face classroom concept of four walls, a ceiling and a floor, is now removed from 
consideration in the use of copyrighted materials. This is an acknowledgement of the reality of remote 
students in dormitories and homes. This important change now allows students to access digital materials 
in a course whenever and wherever they have access to a laptop or a PC. 

3. The storage of copyrighted materials on a server is now allowed, which makes available both synchronous 
and asynchronous use of copyrighted performances and displays [within the limited portions restrictions, 
of course] . 

4. Digitized versions of analog works can be made that are not available in a digital format. In addition, 
digital works that are encrypted may be bypassed using a decryption mechanism, which is a direct 
contradiction of the Digital Millennium Copyright Act. 

5. Faculty, staff and students involved in distance education are now able to legally store temporary, cache 
copies that are made due to the digital process of making digital transmissions. This was accomplished by 
adding §1 12(f) to the ephemeral recording section of the Copyright Law. 

A New Format is Recognized in the Law: “Mediated Instructional Activities” 

The TEACH Act finally recognizes “mediated instructional activities “(MIA). This is a step forward and 
requires that an instructor be present in an accredited non-profit educational institution, not a “diploma mill” or 
profit endeavor. The use of a secure network (password-protected) with software that prevents student copying 
(“downstreaming” — digital copying by students for themselves and others) is also required. For example, it is 
permissible to use a 4-minute clip from Casablanca in a distance education course. Of course, you must use less 
(fewer and shorter clips) than in face-to-face instruction, but transmitting these digital clips is specifically covered 
by the TEACH Act, which was a serious problem before the amendment, since online transmissions (“digital 
networks”) were specifically not included in the old law (Crewes, August 15, 2003). 

What Is A “Session” in the Act and How Does it Apply to My Course? 

Unfortunately, the Act was written as though class sessions occur in a linear manner, instead of in the 
branching pattern of instruction, where all materials are present and students may access all or part of a course 
online at the beginning, as is the case with many online courses (Lutzker, 2002 and Gasaway, 2003). Since there 
seems to be no ready explanation for this approach, “chalk this up” to legislators’ lack of knowledge of how courses 
are conducted in the online environment. However, based on this distinction it is possible to show a clip from Real 
women have curves in a Women’s Studies class “session” (lecture #5) and then have students view this clip as they 
come to it, review this clip as necessary, and also for the instructor to show this clip later in the course in another 
session (lecture #7) if student understanding is found lacking through an exam. 

In September I used a film clip, consistent with TEACH, and I left it on the server for some 
short duration of a "class session." The students can no longer access that session. Today is 
November, and I want to emphasize a point and show once again the same clip. The 
TEACH Act does not bar the reuse of the same clip in the context of a second class session, 
whether you are reinforcing an earlier point or making a new point from the same work. (Crewes, 
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August 4, 2003). 



Can I Use Clips From Educational Videos in My Course? 

No. It is important to note that film clips from videos, dvd’s, and the like are not permissible from 
producers whose main audience is the educational market. This harms their business and is, in the author’s opinion, 
unethical. Copyright was designed to promote the “public good”. Showing clips such as these not only harms the 
producers but also eventually reduces the amount of educational materials available to us all. 

Who is Responsible If Infringement is Found In My Course? 

The institution is actually held more liable than the instructor, according to the TEACH Act, because it is 
the institution’s responsibility to educate faculty, disseminate information on compliance, and to provide secure 

networks for courses (Sections ). On the one hand, instructors can use greater amounts of third-party materials in 

online instruction and their liability for these uses is reduced. On the other hand these provisions focus more on 
institutions, requiring that they develop copyright policies and impose restrictions on access (Lutzker, 2003). 

Can I use copyrighted works as I would in the traditional face-to-face classroom? 

No. Section 110(1) of the U.S. Copyright Act allows more use with the TEACH Act, but it is still less then 
face-to-face instruction. Additionally, the law does bar uses of audiovisual works that might be unlawful copies. 
This statute allows displays and performances. It does not address making copies of any works. Eor that issue, you 
generally need to turn to fair use (Crewes, 2003). 

Can I Use articles that I would Normally Place on reserve at the library? 

No. As previously mentioned, you are limited to less than in a live classroom in several instances. Link to 
the article. Better yet, give students choices of the articles that they may use and/or link to instead. It’s better 
pedagogy, in that it requires students to use higher order thinking skills of analysis, synthesis, and evaluation when 
they search, select, analyze and evaluate articles on their own, and also makes students problem solve about what 
articles would best fit the assignment given. An added “bonus” is that the instructor benefits from this student 
research by continually refreshing her/his knowledge base with each new class. With the “hotlinking” capabilities 
of Blackboard and other online course utilities, students can automatically link to an article or click on Google or 
other search utilities to find similar articles, perhaps finding newer ones than the instructor did when conceiving of 
the course even a few months earlier. The author believes that these capabilities make many of these use issues 
moot. In the event that an article is absolutely necessary and is only available in print, then reserve is a necessary 
option. Displaying a portion of an article is also an option with the “fair use” exemption to illustrate a particular 
point. 

Can I Link to Articles or Other Sites? 

Generally, yes. Simple linking to authorized sites (like a public website) is not a copyright violation, unless 
you try to “deep link” (make the other site look like it’s a part of your own) (Talab, 2000). Occasionally, a web 
author will request that no one link to a site without express permission. While the merit and legality of this request 
on such a world-wide publishing platform as the web is questionable, the instructor should honor it. Anyone who 
places this type of article or material on the web either has no secure “home site” to use for a presentation or for 
other purposes or lacks an understanding of the nature of the web and the fundamental benefits of recognition 
through the citation model. In other words, there is benefit to the old saying, “ I don’t care what you say, just spell 
my name right” that “oils the wheels” of academia .Additionally, “libraries should negotiate their database licenses 
carefully to be sure that they do not attempt to prohibit such linking” (Dolvak, 2003). 

There are other concerns. The site may have questionable legitimacy. Eor example, a site may be a 
“front” for a political action party or other entity with a reason to present information in such a way as to draw a 
conclusion which is not based on fact. In short, they may “have an axe to grind”. Eor example, a site on any 
controversional , political, or religious topic is ripe for such a use. A way to check the validity of a site can be done 
with a PC. Look to see what the host site is at the lower left bottom toolbar while the site URL is being accessed. 

If the site is managed by a biased group the site’s host URL can be seen, indicating the basic approach to or reason 
for the site. Otherwise, it is best to link to related sites, if any are provided , that may indicate a site’s bias. 

Can I Show A Motion Picture DVD Clip? 

Yes, if you can break the copy lock. A Princeton graduate student learned that depressing the shift key 
while the dvd was loading would defeat the copy mechanism, which caused the stock of the company to decrease by 
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ten million dollars in one day (Foster, October 13, 2003). However, in practice, it is quite difficult to break the copy 
protection to get a clip. The TEACH Act allows the bypassing any encryption in digital works for limited portions. 
This is a problem with the law. The Digital Millennium Copyright Act [DMCA states that encyrption breaking is a 
potential criminal violation of the DMCA. The TEACH Act contradicts the DMCA and allows instructors to bypass 
encryption, a fact that the Consortium of College and University Media Centers drew attention to this year 
(Camavale, March 28, 2003). This group was also responsible for developing the Educational Multimedia Eair Use 
Guidelines. 

Does The TEACH Act Provide Enough Options for My Distance Education Course? 

No. The best solution is to make judicious use of all options, listed above, to ensure that adequate 
materials are available to teach your course . This, in itself, requires a fair amount of problem solving. However, 
innovation is the hallmark of countries that utilize the concept of intellectual property protection. It’s worth it. 

TEACH Act Don’ts, Do’s, and “Tried and Trues” 

Do not: 

digitize an entire literary dramatic work 
digitize an entire dramatic musical work 
digitize coursepaks 

digitize or use digitized materials intended for the educational market 
Do: 

use “reasonable and limited portions” of audiovisual works 

use “reasonable and limited portions” of dramatic musical works 

use an entire NON-dramatic literary work 

use an entire NON-dramatic musical work 

link to a site rather than copying or digitizing the article 

Tried and True: 

Use the TEACH Act provisions, fair use, and the Educational Multimedia Fair Use Guidelines 
together 

Teach students to problem solve and navigate the web by finding articles on a topic 
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Peer Coaching: A Faculty Development Strategy to Improve Online 

Instruction 

Shauna Tonkin 
Jason D. Baker 
Regent University 

Teaching in Unfamiliar Territory 

By the time someone becomes a teacher, regardless of whether he or she enters the K-12 school system, higher 
education arena, or corporate training environment, he or she has literally had decades of experience with face-to- 
face instruction. While new teachers vary in their pedagogical training and student teaching experience, they still 
benefit from a lifetime of experience as students themselves. Accordingly, most of today’s teachers have a fairly 
common set of experiences and expectations to draw from when planning and evaluating traditional instruction. 

This is not so when the educational environment is shifted from the four-walled classroom to the online Internet 
environment. According to Bork (2002), the results of a survey of university instructors revealed that experienced 
online instructors had taught between four and seven online courses either partially or fully online. While not an 
insignificant number, it pales in comparison to the teaching experience of the same respondents (e.g., 36% having 
more than 20 years of experience and 34% with 10-20 years of experience), as well as the non-teaching history that 
such instructors invariably had. 

Moving into the comparably new territory of online teaching and learning thus requires a renewed emphasis on 
training and continual improvement. In their 2000 report Quality on the Line: Benchmarks for Success in Internet- 
Based Distance Education, the researchers at The Institute for Higher Education Policy (IHEP) identified twenty 
four benchmarks for ensuring effective online education. Among the faculty support benchmarks were the 
following: 

• Eaculty members are assisted in the transition from classroom teaching to online instruction and are 
assessed during the process. 

• Instructor training and assistance, including peer mentoring, continues through the progression of the online 
course. 

Reliance upon student and faculty self-reporting instruments as the sole forms of evaluation are particularly 
problematic since the online learning environment is significantly less familiar to participants than the classroom. 
Such unfamiliarity is likely to result in feedback which by itself is ill-suited to meeting these quality benchmarks. 
Incorporating peer mentoring or coaching into the instructors’ training and support plan may enhance initial 
experiences with the online environment, which may contribute to improved teaching effectiveness. Eurthermore, 
the use of peer coaching as a means of regular professional development will encourage faculty toward continual 
improvement of their online pedagogical strategies in light of increasing student and technological sophistication. 

Peer observation and coaching activities are an accepted means of generating data for assessing teaching 
effectiveness; furthermore, the practice reinforces the concept that faculty are the best judges of institutional quality 
(DeZure, 1999). However, in the absence of established procedures and adequate training, peer coaching is benign 
at best and in many cases wastes valuable faculty resources. 

Peer Coaching Cycle 

Regent University has employed a method of peer coaching for face-to-face courses throughout the past six 
years. This method is taught in a year-long teaching improvement seminar. Participants engage in a coaching cycle 
with another colleague, sharing their findings with the larger group when the process has been completed. A team 
of experienced online instructors is currently adapting this peer coaching model for the online environment and has 
performed preliminary online peer coaching during this past academic year. 

The peer coaching cycle used for face-to-face instruction consists of three stages: a planning conference, 
instructional observation, and a reflecting conference. The planning conference is a structured meeting in which the 
instructor and his/her peer coach discuss their collaborative effort with a particular focus on the goals of the 
instructor. Unlike other observation and feedback models, this peer coaching model does not promote open-ended 
feedback. Rather, during the planning conference the instructor briefs the peer coach on the specifics of the 
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upcoming class session including such factors as the topic under consideration, the objectives for the session, the 
planned teaching and learning activities, the sequence and pacing of the session, and formal or informal assessments 
included in the session. With this background established, the instructor requests specific aspects that he/she would 
like the peer coach to observe. For example, the instructor may have difficultly sustaining a robust class discussion 
and would like the peer coach to observe the discussion dynamics and offer recommendations for improvement. It’s 
important that the instructor have a clear purpose for engaging in peer coaching and can communicate such requests 
to the peer coach during the planning conference especially since feedback on topics other than these pre-arranged 
ones is generally off-limits. Such focused requests help to build trust between the instructor and coach, ensure that 
the observation and feedback will support the instructor’s goals, and promotes genuineness and vulnerability (rather 
than an instructional “performance”) during the lesson. 

The second phase in the peer coaching cycle is the actual instructional observation. For face-to-face 
classes, we expect that the peer coach will spend between 45 and 90 minutes observing the selected class. The peer 
coach takes notes based on his/her observations of the class, using the planning conference requests as a framework. 
In addition, there are many observations which we encourage the peer coach to make, even if he/she doesn’t directly 
communicate the findings to the instructor. The classroom environment, instructor placement and movement, 
student reactions, questions posed to the class (e.g., type, frequency counts, and even time between question and 
answer), interaction patterns, instructor mannerisms, content knowledge, visual aids, and use of class time are all 
facets which the peer coach could use to assist in the evaluation of the instructor’s requests. Following the 
observation exercise, the peer coach is encouraged to review and organize his/her notes in preparation for the 
follow-up conference 

The final phase is the reflecting or post-observation conference in which the instructor and peer coach meet 
to debrief the observed class session. During this meeting, the peer coach begins by providing the instructor with 
feedback based on his/her observations of the class. The feedback presentation typically includes describing the 
relevant observations, analyzing and interpreting the observations in light of the questions posed during the pre- 
observation meeting, and offering additional data which support the focus questions. After presenting such feedback 
to the instructor in an informational (vs. judgmental) manner, the peer coach elicits the instructor’s inferences, 
opinions, and feelings. This provides an opportunity for the pair to dialogue about the observations and their 
consequences rather than having the peer coach simply debrief the instructor. The peer coach is discouraged from 
giving direct advice but instead to listen intently, ask clarifying questions, focus on the specific observations rather 
than offer personal commentary, and seek to agree together as to the meaning of the observations. The peer coach 
then closes the conference on a positive and productive note by helping the instructor develop an action plan to 
improve in the focus areas. 

As a postscript to the peer coaching process, we have encouraged peer coaches to document their relevant 
observations in an essay or letter to the instructor as a means of helping the instructor use the experience for 
significant professional development. Furthermore, subsequent to the reflecting conference, the instructor is asked 
to comment on the collaborative coaching process (preferably in writing). This provides a final opportunity for 
reflection by the instructor and also helps to provide feedback which can be used to improve the peer coaching 
process. 

Although the online peer coaching process is still under development, our initial efforts have largely been 
to adapt the three-phase cycle to the online learning environment. This has required some modification of the 
procedures to accommodate the uniqueness of online learning. For example, rather than the peer coach observing a 
face-to-face class session for 45-90 minutes, we recommend that the peer coach log into the Blackboard course site 
multiple times during the course of a week. Since the majority of our online classes follow a traditional semester 
schedule but break the lessons into week-long intervals (highlighted by asynchronous threaded discussions), one 
week in Blackboard is equivalent to a week’s worth of face-to-face class time. In addition, we encourage the online 
peer coach to take particular notice of the virtual classroom environment and interpersonal communication 
dynamics. Such facets include the design and layout of the Blackboard Web pages, the tone of the announcements 
and course materials, the level of learner-instructor and learner-learner engagement in class discussions, the types of 
media used for presenting materials, the ease of navigation, the clarity of course instructions, and the instructor’s 
mastery of the course content and effectiveness at presenting it to the class. 

Peer Coaching Checklist 

The collaborative coaching model can be presented in a checklist format as follows: 

Pre-Observation Meeting 

1 . Identify the instructor’ s concern about instruction 
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2. Translate the concerns into observable behavior 

3. Identify procedures for improving the instructor’s teaching and students’ learning 

4. Setting goals and content, arranging time for observation, and choosing appropriate instruments 

Online Classroom Observation 

5a. Class online atmosphere 

5b. Learner-instructor interaction 

5c. Learner-learner interaction 

5d. Mastery of content 

5e. Manner of presentation 

5f. Media for presentation 

5g. Use of Blackboard and the Internet 

5h. Design usability 

Post-Observation Meeting 

6. Provide the instructor with feedback 

7. Elicit instructor’s inferences, opinions, and feelings 

8. Close conference on a positive and productive note 

9. Evaluate the process 
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Abstract 

An examination Asian pedagogy and its relationship to instructional technology over the past five years, 
particularly the impact it has on Asian students who come to the United States to pursue studies in the field. A 
review of the findings sets the stage for what comes next - modifying the pedagogy model employed in Asia. Is such 
action possible? Should it be pursued? What are the consequences to those who teach, those who learn? The 
findings of the research team, discussed and illustrated through selected video excerpts, validate the original 
analysis of the problem - that culture impacts performance. Lessons and UCF model are suggested. 

Introduction 

Since the 1999 Denver AECT conference, a research team has been examining the impact on students from 
China and Taiwan when they come to the United States to study in instructional technology graduate programs 
(Cornell & others, 2002). The research originated following a series of observations as to specific behaviors 
exhibited by these students when in class. These behaviors, while not exclusive to Asian students (American and 
other international students often behaved in similar ways), reflected Asian communication patterns that created 
concern among their professors. The concerns related to student silence in face-to-face classes, accompanied by 
little significant participation online when such was required by their professors. Based on the information obtained 
from all subject groups, we designed a series of interventions to address the problems cited, at least in part. These 
included pedagogical strategies of a general nature and those targeted at students in the instructional technology 
graduate program. A series of video modules, produced in-house, related to assertiveness and accompanied an on- 
line print module. Especially interesting to us was that increasingly, the use of technology in instruction had been 
growing at what many would say is an unparalleled rate across Asia. We wanted to affirm through live video 
interviews, that when technology is employed within the context of Confucian-based learning, the role of both 
teacher and student changes drastically. Would Asian educational institutions seeking to inject their teaching with 
technology be able to seamlessly make the change? What roles would teachers have to take in this new techno- 
world? How would students react and how would teachers evaluate their performance? 

Conclusions reached point to yet another problem, one of major proportions, that may or not be easily 
solved - the modification of extent Asian pedagogy that recognizes the changing role of the teacher/professor when 
technology use is integral to content mastery. It is our belief that, with technology in the hands of the student, the 
teacher/professor is no longer the locus of control. Such a change in authoritarian role may well be anathema to 
traditional Asian educators. Indeed, the Confucian model may well be under attack, at least in part, and this is the 
nexus of our concern. How can an educational system steeped in Confucian tradition of 2500 years or more, survive 
the onslaught of the technological revolution within the schools, be they at the primary level or within the colleges 
and university across Asia? Something has to change. . .or does it? Is there room for reconciliation herein 
(Trompenaars and Hampden-Turner, 1997)? 

In the past few years, both China and Taiwan have initiated legislative mandates flowing from their 
respective Ministries of Education that, henceforth, both English as a Second Language and Computing will become 
core subjects of study from the primary grades onward. Other Ministries of Education across much of Asia have 
initiated similar mandates or at least moved along similar pathways. 

A recent Beijing visit revealed that a new technology initiative aimed at the Western Provinces is in the 
making, one that will offer three variations of distributed learning via l)E-mail; 2) satellite video feeds augmented 
by e-mail; and 3) Web-based instruction (Liu, Y.Q., personal communication, September 17, 2003). Western 
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provinces were targeted because those in the Eastern provinces generally enjoy a higher standard of living and also 
considerably more financial support provided within those schools. While complete details about this initiative 
remain sketchy, the findings of the committee tasked by the Ministry of Education should be forthcoming near the 
first of 2004. What is not so apparent, either through discussions with Ministerial officials or through a review of 
the literature is how such initiatives will be implemented and the potential impact such initiatives may have on both 
those who teach and those who are their students. The Confucian model holds dear the role of teacher as being 
elevated above all else within a school. This same model dictates that the teacher is the ultimate master of the 
content taught. He or she is also the sole authority within the classroom, both in terms of what to teach and how as 
well as in matters related to personal conduct, philosophy, and values. Into this picture of authoritarian hierarchy 
now comes technology, and with it, an unaccustomed freedom by students to use technology towards their own 
educational ends. When technology is used in class, the teacher is no longer the ultimate authority; no longer is he 
or she the master of all knowledge and belief. Indeed, the teacher’ s role has changed, congruent with much now 
modeled in the West. This is akin to revolution, not evolution, and the impact of such massive change, be it implied 
or acted upon independent of what the teacher says or does, has major consequences on Asian pedagogy in the 
future. That future is now! 

In an effort to seek renewal and fulfillment, we under took the task of trying to stem a potential tide of 
misunderstanding by those ultimately charged with carrying out the mandates imposed upon them in both Taiwan 
and China - its teachers and professors. We originally referred to the plight of international students coming from 
his or her country to another for study, finding that nothing is as it should be. Such a situation pales in comparison 
to the chaos that might be expected when we take the West to the East. The background Asian students bring to the 
West reflects an educational culture that saw its birth thousands of years ago, well before European and American 
scholars developed their educational theories. Plato (428-348 B.C.) and Socrates (470-399 B.C.) come closest as 
peers to Confucius (551-479 B.C) (Beck, 2002) for it is he, and his contemporaries who shaped the way education is 
currently practiced by well over one billion people. Or put another way, over a third of the world’s total population. 

Our initial work traced the pedagogical roots of a specific group of international students (all of whom were 
from either China or Taiwan) who came to one university, the University of Central Elorida in Orlando, and enrolled 
in one graduate program, the Master’s degree in Instructional Systems (http://pegasus.cc. ucf.edu/~instsys/,2002). 

Its premise was (and is) that 1). Culture does make a difference, especially when two cultures have the 
possibility of engagement and that 2). These cultures are generally 180 degrees opposite to one another, sometimes 
while on a near-collision course. 



Statement of the Problem 

We expect that Instructional Systems graduates develop competencies that allow them to communicate to a 
variety of audiences, both verbally and in writing. They must be able to deliver presentations using a wide array of 
technologies. They must write papers that reflect both eloquence and conviction. They must work well as a 
member of a multi-cultural team and they must develop positive assertiveness. These competencies are introduced 
in the introductory Instructional Systems course. Survey of the Application of Instructional Systems (Cornell, 2001) 
and reiterated in Piskurich & Sanders (1998). 

Western students, particularly those in North America, are generally adept at using these skills, given their 
innate tendency toward achievement (judged on what one has accomplished and one’s past record), both 
professionally and socially. Other cultures function more on ascription (status attributed to one by birth, kinship, 
gender, age, one’s connections, and educational record), a phenomena often seen in Southern Europe, Latin 
America, and Asia (Tropenaars & Hampden-Turner, 1998, p. 9). For Asian students, acting on many of the 
competencies found within the Instructional Systems program proved unnatural, uncomfortable, and often induced 
considerable stress. The result - silence, both orally and in written documentation. Could the reverse situation 
evolve when similar competencies are required of Asian teachers? 

The overarching descriptor that guided our approach, we took from two works of Trompenaars & 
Hampden-Turner (1997, 1998), “reconciliation!” It implies neither being dominated by others nor acquiescing but 
rather, “. . .through an integrative process, a universalism that learns its limitations from particular instances, for 
example, and by the individual voluntarily addressing the needs of the larger group (p. 53, 1998).” The “larger 
group” consisted of the Asian students’ Western peers, who numbered a little over half. In the coming years, this 
ratio of Asian-to-Western students may well change, with the numerical balance favoring Asian students! The 
larger group’s needs were not immediately apparent as infusion of Asian students into the Instructional Systems 
master’s program was not a one-shot event but gradual. We recruited one student at a time, until the critical mass 
gradually increased to where it is now at almost half the number of new students in the program. A similar trend is 
already apparent in the doctoral program in Instructional Technology where the critical mass of Asian students 
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exceeds 50% ! All these students from Asia (as well as those from India, Russia, Cyprus, and Turkey), regardless of 
from where they have come, bring with them their own idiosyncratic view of life. It is a pedagogical life that, 
heretofore, has been mostly unknown, by both the American students and their Western professors (2002, 
International Institute). Many professors know little about the basics involved in coming to the United States as an 
international student; they know even less as to the specific logistics; and have even less than less in-depth 
knowledge about the countries from which their international students have arrived! 

Add to this scenario an un-enlightened American student population, both undergraduate and graduate. 

Our North American students are mystified by the silence of their Asian peers. They look to professors to 
encourage their Asian peers to communicate because they, the students, have little success, at least initially. Many 
professors remain perplexed for they also find their Asian students to be mute. How to evaluate muteness - that 
poses their immediate challenge! How to understand the Asian pedagogical culture and the degree of symbiosis 
between that culture and the one found at UCF - that was the challenge! Yet, we continue to insist that these 
students learn and study “our way.” 

Our approach was to seek understanding of what these students bring with them and so, we “went to the 
experts,” professors from both Taiwan and China, and asked them about their formative education years. We asked 
them to describe how learning was for them when they were students, in grades anywhere from within their earliest 
memory of schooling to the time they completed their last degree. In this way, we obtained first-hand descriptions 
that may or may not validate what appears in the literature. However, it is not that we reject what the literature says 
about Asian education; we wanted to add richness and provide increased depth to what we experienced, thus our 
approach was to conduct live video interviews where possible. In this way, we validated major premises found in 
the literature and provided a qualitative database that clearly presents imagery that communicates the contrasts in 
pedagogical styles. East and West. Note, we did not say East vs. West; at least not yet. 

Research Questions 

Are there elements of the valued Confucian pedagogy that might be compatible with those employed in the 
West? Are there elements of Western pedagogy that might be in harmony with Confucian pedagogies? Are there 
implications that this research might have for students from other cultures? Are the findings of value to technology- 
based commercial organizations who engage in multinational commerce, especially given the current and projected 
increase in such initiatives in East Asia? In addition, is there symbiosis or asymbiosis - that is the goal we seek to 
address? Reconciliation between the cultures will be out outcome. 



Brief Review of the Literature 

Even a cursory review of the literature suggests images that stereotype our students, be they from China, 
Taiwan, or North America. It is tempting to label our Asian students as being passive, quiet, silent, submissive, 
respectful, intelligent, hard working, introspective, disciplined, and many other descriptors and, for some, these are 
accurate in their assessment (Tu, 2001). Hampden-Turner and Trompenaars (1997, op cit., p. 10) identify values 
differences between the West and East Asia in the business sense as having differing historical roots. 

West East 

Supernatural religion Secular humanism and enlightenment 

Belief and faith Paradigmatic assumptions 

Cartesian dualism The Way of Complimentarity 

Values as things Values as wave-forms 

Cultures and Values -Yin Cultures and Values - Yang 

Pioneer capitalism Catch-up capitalism 

Finite Games Infinite Games 



We include these values sets because there is a continuing relationship between instructional systems 
designer competencies expected in the West by business and industry and the extent conditions the Asian students 
bring with them. This relationship often reflects the Yin and Yang depicted above. Hampden-Tumer and 
Trompenaars define belief differences, those in the West being anchored in “. . .the commandments of supernatural 
beings and their sanctions for good or bad behavior in an after-life”, (p. 10-11). In East Asia the authors cite value 
systems that, while varied across different countries, are based on “their wisdom being secular, practical and of this 
world.” (p. 1 1). That “they are ‘humanistic’ in the sense that they aspire to improve the human and social 
condition, and failure in this leads to criticism and reappraisal, (p. 1 1). 
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In the literature, we found ample evidence of how things are. We found little of help as to what to do about 
it, especially as relates to ways in which we might make the lives of the Chinese students in our program more 
positive. We identified a number of categories wherein we gained insight as to what beliefs the students bring with 
them. As Trompenaars and Hampden-Turner assert, however, “you can never understand other cultures (1998, p.l) 
and so, while it is our intent to provide clarity as to Asian thinking, understanding might lie beyond our real 
capabilities. The best we can hope for is to create sensitivity toward the problems raised, and in doing so, perhaps 
emerge with observations and strategies that may make life for our students far better. 

In support of this caveat is a statement read from another source, now disremembered (Anonymous, N.D.), 
that described the feelings of an American expatriate in China who had worked there for more than 20 years. A new 
arrival to China asked him something to the effect of, “With all your years living in China, how long did it take you 
to understand the Chinese?” His replied, “I still do not understand them and probably never will.” He added, “One 
cannot understand those from another culture unless they have been born or grown up with it!” So much for really 
understanding our Chinese students in Orlando. 

In terms of what is valued and how it is manifested within the UCF classroom, the literature reinforces our 
own experiences. For example. An ongoing concern is the silence of Chinese students in our classes, especially 
during their first or second semesters with us. Is silence always “bad?” This question was posed us at an Orlando 
conference of the Comparative and International Education Society (CIES) by Dr. Haiyan Hua, Senior Research 
Associate of Harvard University’s International Education Group. We were quick to respond in the negative, that 
no, there are many times when silence is, in fact, “golden.” 

We found evidence of why our Chinese students were silent in Brooks (1997) who was quoting Yum 
(1994); ‘ . . .the Chinese communication process places the ‘emphasis . . .on the receiver and listening rather than the 
sender or speech making’ “(p.83). Brooks emphasizes Yum’s point that Chinese are receivers of messages, listening, 
rather than being senders. She provides a pedagogical blueprint as to the nature of how Chinese students are 
educated; 

Confucius believed that. . .a hierarchal system was essential to the harmonious well being of society. This, 
in turn, is reflected in the Chinese classroom. Chinese students regard their teacher as all knowing, and the 
absolute authority on the subject matter. . . .Chinese teachers are under severe pressures not to make 
mistakes, not to misguide students, and not to be criticized. . ..It is the duty of students to give utmost 
respect to the teacher. To ask questions of the teacher. . .is tantamount to questioning the position of the 
teacher, and therefore is not a feature of Chinese classrooms. Since the teacher is the sole authority in the 
classroom, rigid order and formality are the main features of the Chinese learning environment (Su, 1995). 
Zhang, Sillitoe and Webb (1999), in quoting Ballard & Clanchy (1991) compare Western and Chinese differences in 
the attitudes toward learning; 

“. . .in Western culture, tertiary education is oriented towards extending knowledge. Therefore, the teaching 
approach used and learning approach encouraged are designed to develop analytical and speculative 
abilities of students.... the [Chinese] education systems are mainly oriented towards conserving knowledge, 
and the learning approach fostered emphasizes the reproductive ability of students. 

Tu (1999) reinforces these findings by stating that Chinese teachers have been accustomed to teaching where there 
was only one-way communication and a quiet environment. This no longer exists. All subjects are discussed, even 
the teacher’s private life. Teachers fear the loss of their authority and the pressure of public opinion (p. 3). He (Tu, 
2001) adds that Chinese collect information from non-verbal channels and perceive more exact information than has 
been delivered (p. 5). Lin and Yi (1997) add more pieces to our puzzle when they describe further issues faced by 
Chinese students that compliment those already discussed; 

International students from Asian countries are often stereotyped as quiet, reserved and non-assertive. 

These cultures place an emphasis on harmony and respect for authorities. Therefore, many of them are 
reluctant to share their feelings or emotions, express their opinions or oppositions to anyone, especially to 
authority figures.... Asians tend to emphasize the importance of patience, harmony, respect and deference. 
Asian cultures tend to place a high value on team efforts of collectivity. . . .Asians are also modest about 
their accomplishments. Many Asian international students feel uncomfortable with the individualism and 
the competitiveness associated with the American culture. 

This is not to suggest a caveat, a series of excuses to fall back upon; rather, it is a reflection of reality for many. We 
cannot stereotype our students so easily, be they from Asia or Central Elorida, having had students from both these 
sets of stereotypes. The common denominator, metaphorical in many respects, seems to be what was termed the 
“figure -ground” relationship (McLuhan 1964), i.e., that students will perform within parameters that are related to 
their most comfortable environment, whatever suits them at the time. In “McLuhanese” this concept is expressed as; 
'Eigure' refers to something that jumps out at us, something that grabs our attention. 'Ground' refers to something 
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that supports or contextualizes a situation, and is usually an area of Mnattention (Gow, 1998). Mercurial in many 
ways, chameleon-like in adaptability - it’s just what students do to survive! 

The literature excerpts provided are but a tip of the pedagogical iceberg. It is our intent to continue 
surveying the literature continually to better gain understanding of our own Chinese students. Meanwhile, we are in 
the process of developing a methodology that will enable us to better understand the implications implicit in 
teaching Chinese students at UCF. We think that our findings may also prove useful to other North American 
institutions wherein Instructional Technology programs are offered. Ultimately, we envision the results we obtain as 
also being of considerable use to multinational corporations, many of whom hire many Asian employees. An initial 
description of our methodology follows. 

We realize that what we include in this article, as well as in our presentations, may appear offensive to 
some, to marginalize some, to be found guilty of being patronizing to others. . .none of which is intended. We are 
taking observations as we get them, subjecting them to analysis, comparing them to other research found in the 
literature, and then attempting to devise useful strategies that will enable our Chinese and Taiwanese students to not 
only succeed, but to prosper! In effect, we are trying to level the academic playing field at the University of Central 
Florida so that there is equity for most, if not all. 

What we were trying to do 

The genesis of this study began a few years ago after the last author, having recruited numerous students 
from Taiwan and China into the Instructional Systems master’s program at UCF, found that, while each student 
brimmed with enthusiasm, when it came time for communicating ideas in class, most remained silent. The students 
were always pleasant, respectful beyond belief, and had a work ethic akin to our forefathers. In class, when asked a 
question directly (as is typical in North American classrooms), they would look down at their notes, mumble soft 
words, most of which were unintelligible, and squirm uncomfortably in their seats, knowing that multiple eyes were 
upon them. When asked to post messages on an online discussion forum, the most one could anticipate from our 
Asian students was a perfunctory few words of acknowledgement, “Yes, I agree with her statement’’ or similar 
responses. Not much to go on as an evaluative tool. Then there were the infamous synchronous chat-rooms. . .where 
written paralysis was the rule for our Asian students, not the exception. The odds of the professor really knowing 
what the student knew were getting slimmer by the moment. 

Another caveat: The instructor in this instance was one who evaluated performance based on what the 
students do, tied to performance objectives and goals that, more often than not, include project-based assignments, 
rather than conventional paper and pencil objective tests. In addition to individual and group assignments, emphasis 
is placed on participation, on the development and continuation of positive attitudes, of being self-motivated rather 
than having to be told what page to read or how many words were necessary in a paper to receive an A. He placed 
continual emphasis on teamwork, oral and written communication, as opposed to memorization of facts and figures. 
In short, his pedagogical model was 180 degrees opposite that typically followed by his Asian students! It was and 
is all about effective communication. 



Methodology 

It was decided to interview four separate and distinct groups of individuals, both composed of either 
teachers/professors or students from China and Taiwan. 

By using these individuals as SME’s (subject matter experts), we were authentic in our selection of subject 
class. We then needed to differentiate between the groups to provide focus on differences that might exist between 
them. 

We decided to use four groups of ten, an N of 40, consisting of the following: 

1. Teachers/professors who had been educated in Taiwan or China and had not studied abroad. We 
anticipated that the demographics of this group might be those who were more experienced and whose 
teaching would mirror much of the methodology they learned from their own teachers. 

2. Teachers/professors educated in Taiwan or China who had completed additional study in the West, be 
it in North America, Europe, Australia, New Zealand, etc. and had since returned to teach in their 
homeland. 

3. Professors currently teaching at the University of Central Florida from China or Taiwan. 

4. Students from China or Taiwan studying at the University of Central Florida, preferably those enrolled 
in graduate Instructional Technology programs. 
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Interviewee Questions 

The researchers established two basic questions to which they sought answers from the Chinese and Taiwanese 
subjects. The subjects in three groups were teachers or professors from China and Taiwan. The fourth group was 
comprised of UCF students enrolled in the Instructional Systems/Instructional Technology graduate program areas. 

The questions were: 

1 . When you were going to school (meaning any time between your primary school and graduate 
education), how were you taught and what were the recurring values that your teachers instilled in 
you? 

2. Now that you are a teacher/professor, what strategies do you use when you teach and what values do 
you feel are important to instill in your students? 

2a. A fourth group of both Chinese and Taiwanese was composed of current students enrolled at the 

University of Central Florida. In addition to question 1, above, we asked, “How are the instruction and the 

values you are learning here at UCF different than that you received at home?” 

What we were (and are) trying to do is to get a clear picture of how our Chinese and Taiwanese students learned and 
what values they bring with them when they come to UCF. 

Each subject was asked the questions posed while being recorded on video. The sessions were relatively 
brief, given that there were only two basic questions being asked. The subjects could reply in either Chinese 
(Mandarin) or English. If Mandarin was selected, the interviewer spoke Mandarin as well. 

Transcripts of each interview were produced and analyzed. An analysis was made of key words across all 
groups and recorded in terms of similarity, frequency of use, and by category of interviewee. A series of figures was 
constructed that depicts the relationships between key words used and group type, thus profiling group differences in 
a more visible manner. An edited tape of relevant excerpts was abstracted from the raw footage that highlights the 
results of the taped interviews. 

With the analysis of Asian (Confucian) pedagogical methods validated, efforts will be made to affect a 
comparison between these methodologies and those regularly employed within Western university classrooms, more 
specifically, those wherein Instructional Systems design students are enrolled. The reverse analysis is used to 
examine issues related to infusion of technology in Asia. 

Preliminary Results 

Confucian pedagogical influences upon Chinese and Taiwanese professors/teachers and their students 

This is a longitudinal study in that, as time and financial resources permit, the data collection will continue 
until there is a numerical balance in intra-group findings. Eollowing is a sampling of some of the interviews 
conducted thus far with the teachers and professors. They are summary statements arranged according to categories 
that will eventually appear in the matrices being developed. Reactions from students follow. 

What it was like when I was a student is included in the first section and key elements were emerged and 
identified. These key elements were compared among the group 1, group2 and group3. They are behaviors 
disciplines, deification of the teacher, learning objectives, interaction, educational media, and a strict classroom 
environment. When interviewees talked what it was like when they were students, they described how they were 
taught and how they learned. Comparing how they were taught and how they learned, there were some conflicts on 
behaviors disciplines, deification of the teacher, learning objectives and interaction. 

Behavior and discipline 

Group 1 and Group2 (faculties with/without overseas experience) both expressed that they were taught to 
follow the traditional Chinese Confucian way of being a student. Being a student in Confucian way, the silence was 
addressed by lots of teachers and professors in the first two groups. They were taught to “Be a good student, a good 
son and a good daughter. Must have excellent behavior. ” “ Teachers teach us to be quiet, keep quiet, don ’t make 
noise, you just listen. ” “ Be humble and do not speak too much. Just listen. ” “Silence is god. Don ’t talk too 
much. ” By being taught this way, group 1 and group2 were more conformed than the third groups. UCE professors, 
as the third group, emphasized their feeling against such Confucian behavior disciplines. When they are talking 
about the Confucian students’ behaviors disciplines, UCE professors used ironic tones. “lam afraid of them 
(teachers). They are horrible. They did whatever they want. ” “They didn’t complain whatever the professor asked 
them to do ” 
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Deification of the teacher 

The role of a teacher was deified and amplified due to its tradition (i.e., Confucius teachings). Teachers 
were treated as an authority in knowledge, values, and life experiences. All three group faculties shared the 
perspectives regarding to teacher’s role. They expressed that “teachers also play a high authority role. They 
introduced the knowledge and how to prepare for the exam”. Teachers and professors are like fathers” “Teacher is 
the authority”, “All that teachers said is the truth. ” “Social-status of teachers was very high” 

Regardless of how sufficient their knowledge and skills may have been, individuals were taught to be 
obedient and humble before teachers and they were required to look up to their teachers. All the faculties in the 
three groups respected their teachers very much. “My parents asked me to: listen and learn, just be quiet, don ’t 
make trouble. ” “They are our role models” “We always admire our teacher” “we really respect teachers”. “The 
teacher played a high authority role. ” 

As a result, teachers were among the top of the social hierarchy. Their teachings were like truth. Such high 
authority influenced students in many ways. One teacher in Taiwan said that “I respected my teachers, and I had a 
dream of being a teacher as a child. They influenced me so much, so I decided to become a teacher. ” In another 
way, most faculties expressed their conformation to such high authority without question. “I dared not say anything 
against them. Their teaching method influenced me a lot. ” “What teachers said was right. I didn’t fight against 
them. ” “You would respect to them whatever they said and whatever they did was up to them. ” “I thought of my 
teachers as persons of high social status. I couldn’t challenge them” 

Learning objectives 

All three groups of teachers and professors expressed their dissatisfaction towards the way their teachers 
taught them. When they were students, they are not sure of the reason to study so hard. The objective of learning 
seemed only for the exams. Under the exam-driven teaching methods and environment, their teachers did not relate 
their teaching and the real world. “No one told me why I should do it, I was just asked to do it”. “I didn ’t know why 
to learn this stuff as a result”; “We have to study hard. We have to work hard. We have to prepare many times to 
pass the exam. ”, “Pass entrance exam to study at a higher level. ” “Prepare for college entrance exam, work really 
really hard. ” “Pass the exam to survive to next level, from middle school, to high school, then to college. ” “Study 
hard and got good grades in the examination ” “The goal was passing the exams ” “Didn ’t work well to bridge 
knowledge and the real world. ” “I can not understand the relationship between what I learned in class to the real 
world” Furthermore, UCF professors, based on their teaching experiences, complained that their teachers did not 
give them learning objectives each semester. “We did not know what exactly we would learn during the semester in 
the class. ” “They just followed the subject day-by-day. You don't have the whole picture in your mind. ” 

Interaction 

One-way interaction between the instructor and students was commonly witnessed. Three groups revealed 
a lack of interaction between teachers and students. The relationship between teachers and students is top-down. 
Students felt the distance due to the teachers’ high authority and their teaching methods. Students won’t ask 
questions in class. “When I was a student, I wanted to get close to teachers, but I just dared not. ” “I was afraid to 
discuss with teachers or raise questions in class” “We have very few questions” “We have questions but very few. 
We would go to teacher’s office. ” “I could ask questions of instructors, but they remained superior to me in a 
sense” “We seldom had access to the teacher. ” “It’s not common for you to talk to the teacher after class. Students 
felt distant. ” “Students are afraid to ask dumb questions” “We were afraid to discuss our ideas with the teachers 
or raise questions in class”. “Teachers were not easy to approach” 

All three groups of teachers and professors complained about their teachers’ one-way traditional lecturing 
teaching methods. The instructors lectured and students listened carefully and bore in mind what was told them. 
Interviewees said that “they lectured and we did all what they said for homework. ” “All lectures in every class” 
“Cramming instruction” . “We were sitting there and just listened”. “They didn’t emphasize cooperative learning. 
They don’t involve any activity” “They just use oral teaching strategies. There is no direct response. ” “They 
lectured a lot. ” “No activity at all” “They didn ’t give examples to explain some rules coming from theories ” Based 
on such one way traditional teaching methods, each of the two parties was held responsible for their own tasks. 
Interviewees said that most of the time they were listening and taking notes in class. “Read the textbook for the 
examination. It doesn’t matte if students listen to the teachers or not.” “ We just take notes.” “ We just followed 
the book. ” “ We just follow. If we did something wrong, we had to do homework lots of times” “In terms of study. 
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I just figured it out on my own" “They blame the student. It’s the student’ s fault if they didn’t do well” “Students 
feel much responsibility for their failure in exams” 

Not only were students asked to be good listeners and note-takers, but it was demanded that they 
retain the information as much as they possibly could. Although it does not sound interactive enough, students did 
not fight against the authorities. “Teachers follow the textbook, to memorize the textbook seriously. ” 

Educational technology 

Most of educational media employed included chalk and board, pictures, and tape recorders. “Teachers 
just used chalk and the black board”. “We recited the text and they asked us to listen to the audio tapes in the class 
three hundred times. ” “Teach by the book. Compare our performance to the tape. Very few recorders were 
available” “Teach by chalk and then do your homework” Since traditional lecturing predominated teacher activities 
one faculty concluded, “Using media to assist was not necessary” . Educational media use was rare; one teacher 
from the second group remembered that his teacher used some live samples to help teach. “He took us to see all the 
live samples around campus, and then he took us to do science practice ” 

Strict Classroom Environment 

From the interviews, most of the teachers and professors are not pleased with how the teacher controlled 
the classroom environment. They all reported strictness within their classrooms when they were taught. “The 
teacher controlled the classroom ” ; “During my studies there, I was taught to be disciplined. I was not allowed to 
be a rascal. I was taught about professionalism. It was close to being military training. ” Teachers need a lot of 
control in classroom. ” “When the teachers walk in. every student has to stand up for respect” 

What it is like now I am a teacher follows and several other major ideas emerged; teacher roles, learning 
objectives, interaction, educational technology, and motivation. There were lots of similarities about how they were 
taught. As to how they are teaching now, there are many differences among the three different groups. 

Teacher Role 

During interviews, teachers and professors without any oversea experience (group 1) liked to talk about 
their roles as teachers. Some wanted to be students’ friends. Some wanted to be students’ mentors. Still others 
wanted to keep their authority roles. “As a teacher, I emphasize knowledge delivery, mental guidance, ” “I feel 
more comfortable when I play a role of a mother as a teacher. ” “I will not allow my students to get too close to me” 
“I don’t play an authority role ” “I am demanding of my students. They do what I tell them to do” “I am not just a 
knowledge dispenser, but a guide of life experiences. I taught them values to conform to society. ” Compared to the 
first group’s diverse views of teachers’ role, groups 2 and 3 faculties did not emphasize their roles as teachers too 
much. This was a very interesting finding. Like one faculty said “I want students like me, and respect me. But I 
don ’t do anything deliberately to make them respect me more. ” Faculties with overseas experience and UCF 
faculties may think the role of “teacher” is not as important in today’s classroom. 

Learning Objectives 

Rather than talking about the role of teacher, Group2 and Group3 faculties emphasized their need to give 
students the big picture. Clarifying learning objectives seems very important for these faculties which reflects 
benefits received during their overseas’ experience. “I would like them to know they need to keep on learning all the 
time for their degree, or after they get the degree. ” “I will try to let them know what’s going on. ” “I introduced the 
students to the whole picture. ” “I always taught my students “you have to keep your mind open so that you can 
learn as much as you can; do not get restrained by whatever the name of the department. ” “They can talk and ask 
you questions and even ask for some of advice about their career. ” 

Interaction 

The lecturing teaching method is still the primary way of knowledge delivery crossing the three groups. 
When the interviewees were talking about their classroom interactions, there are very interesting differences among 
these three groups. Group 1, teachers without oversea experience, tended to emphasize teacher-students 
interactions. “What I do is create more interactions in the activities. It takes more time and effort to prepare 
lessons, but kids like them. We are like friends now. ” “I would like my students to be closer to me than before. ” 
“Positive interaction between me and my students” “I like to share my experience with my students about how to 
use new technology to help each other and to collaborate. ” “We ask students to raise some questions. In this way, 
we have two ways communication ” However one faculty member mentioned “my relationships with students were 
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not really pleasant. ” No matter positive or negative, they focused on the interaction between students and 
themselves. Group2, teachers with overseas experience, are more likely to talk about interaction between students 
themselves, for example group projects, collaborative learning. “I use a lot of group discussion, a lot of team 
work. ” “They would discuss, they would present” “We do some group work, outside reading, gaming and problem 
solving and ask the answer. ” “We separate students in different groups. We let them to teach each other, and share 
the information. ” “They have to discuss their thinking with other senior students or a graduate student in our 
department, or just talk to their instructor or teacher” “I tried to arrange these discussions. In some classes, I tried 
to arrange the seats to give them a comfortable environment ( to facilitate the discussion). ” In addition to the 
interaction between students, teachers in group2 also encouraged students to raise questions. “/ tried to encourage 
the students to ask, I always encourage them to ask, to stand up. ” The teachers with oversea experience probably 
have more strategies to enrich their classroom interaction than those in groupl. As a Western educator, interaction 
in classroom is a very common trend. Without deep understanding of the interaction in the classroom, faculties in 
groupl probably simply apply such trends by increasing the interaction between students and themselves. Beyond 
interaction among students, students and teacher. Group 3, UCF professors even mentioned their customized and 
flexible teaching strategies to meet their students’ needs. “I have group projects, but there are some students who 
just don't like group projects. In those cases I will just use few group projects and then do some individual projects 
because projects are supposed to be harder, bigger problems for student to work on and they can learn by talking to 
each other. They prefer to have group projects. ” All three groups’ faculties like to talk about their ways of improve 
the quality of their courses. 

Educational Technology 

New technologies are increasingly and continually being introduced. Thanks to these innovations, students 
are provided with alternative learning experiences. All three groups mentioned their using technologies to assist 
teaching. It was amazing that, from group 1 to group 3, the discussion about technology got more detailed, and 
negative attitudes toward educational technologies emerged in groups 2 and 3, especially in group 3, the UCF 
professors. Faculties in group 1 without oversea experience, all had positive words about using educational 
technology. However they did not mention how, rather than to just list the very broad categories of new 
technologies. “We use audio visual material that follows teaching, audio tapes, video, VCD, DVD, and computer 
software to teach the students. ” “In my class, it is a new technology environment including multimedia, Internet, 
and so on. ” Faculties in group2 with oversea experience, have more detailed opinions about new educational 
technology, even with some negative opinions on some particular technology, which means that they are really 
experimenting with these technologies to assist their teaching. “I do not like to use transparencies or PowerPoint. ” 
“I tried to use multimedia. Although it is expensive, I still use it. ” “Usually I use my laptop, so I can connect to 
several links. ” UCF faculties discussed their usage of education technologies in a very detailed way. They talked 
about problems that exist in current educational technologies. These problems we perceived as being barriers that 
interfered their teaching. “/ use Internet, course webpage, and use e-mail to interact with students. I put 
assignments on the web; post my homework and exam solutions, and course announcements - everything on the web. 
Also, students interact with me and use email quit a lot. ” “ I told students to clean their email box, otherwise, I got 
lots of rejected emails.” “ I always have a Web site and even a bulletin board where they discuss problems.” “ I 
try to use Microsoft Power Point in teaching, but students didn ’t like it. So I went back to use the board. The reason 
is that use the board is that I can reuse the equation. Those are main parts I use in teaching. The students want to 
have time to understand. They want to follow the flow, follow the procedure how the equation was delivered right. 
There were a couple of semesters I use PPT. When I use PPT, it’s very difficult to show every intermediate step. So 
what happened is that I show the equations on the board and then explain the procedure. But students fall to sleep, 
you know. They didn’t like to respond at all. So I went back to the board. ” 

Motivation 

All three groups’ faculties used different ways of motivating students. Teachers without oversea 
experience focused more on a social and philosophical level. “I taught them values to conform to society. ” 

Teachers with oversea experience focused more on students’ participation in classroom. “I tried to encourage the 
students to ask, I always encourage them to ask, to stand up. ” “I asked them to use methods we have taught during 
the class. They can pick whatever topic that interests them. ” “I ask students to prepare something to report and to 
comment on these reports. ” For UCF professors, in addition to encouraging students’ participation, they provided 
more error-proofing motivation which the other two groups did not mention at all. “I can ’t make them lose face. I 
told them ” you can try more next time. ” “I have to encourage them instead of blaming them. Tell them screw-up one 
exam that’s fine, try next time. ” “You can come to my office any time as a student. My door is always open. ” 
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Western influences (i.e., syllabi) become salient in instructional pedagogy of the East in this case. 
Teachers measured student performance at both individual and team levels. Collaboration and coordination were 
foreign to the culture of long ago, but even now, the antecedents of that ancient culture remain deeply immersed in 
the new strategies. We hope that further research in these areas will bring together the values of both cultures, for 
we must nurture both. 



Reactions from the UCF Students 

Rather than ask our students how they changed when they became teachers, we asked them, “How are the 
instruction and the values you are learning at UCF different than that you received at home?” 

Generally, the student responses reflected both positive and negative statements. Their replies to question 1 
paralleled much of what the teachers and professors across all three groups said. Their answer to Question 3 
seemed, as much as we could sense, a reflection of both their Confucian academic background and the degree to 
which they welcomed change. 

Those students more open to new ideas and experiences delighted in the differences they found between 
Eastern and Western pedagogies. They were also more critical of their Western experiences, in a positive sense, 
articulating concerns as to variations between their various professors’ teaching styles. 

Most in both student groups found the intensive interactivity employed in class initially baffling but 
subsequently empowering. The more traditional students decried the perceived “lack of specific direction” to which 
they had become accustomed while those more inclined toward change welcomed it, recognizing that its intent was 
to foster self-initiative and motivation. 

Since they were all instructional systems majors, they welcomed the plethora of technology use while, at 
the same time, questioned its use in terms of appropriateness by some professors. 

Summary and Conclusions 

A reflection of Confucian pedagogy emerged across all groups and its importance reflects the current 
environment wherein the subjects are located. A value set emerged across groups that reflect Confucian pedagogy 
with a consistent series of key words that identify specific teaching and learning attributes. The words and phrases 
we have included are a partial reflection of the data analyzed. The knowledge gained from the interviews provided a 
past, present and most likely future portrait of the relationship between Confucian pedagogy and the needs of our 
UCF Chinese students. The conclusions reached have parallel meanings for other students from Africa, Europe and 
Latin American wherein similar pedagogical hierarchies exist. The research provided the basis for proceeding with 
Stage 11 of the research, assisting teachers in China and Taiwan in adapting to Ministry of Education edicts that 
English as a Second Language and computer competencies be taught in the public schools. Indeed, perhaps we can 
nudge Alishan Mountain. . .a little bit at a time. Who knows. ..even the Great Wall may also experience tremors. 



Significance and Need for Further Study 

We view this research as being in its infant stages, with years of further work ahead of us. We are not 
unmindful of the sensitivity of our undertaking. We are especially concerned that our actions do not “change these 
Chinese students to the extent that, upon their return to their home environments, their effectiveness has been 
culturally emasculated” (Ying, N. personal communication, April 4, 2002). 

Conversely, Chin-Ning Chu, in his work. Thick Face, Black Heart (1992), commenting on how one follows 
rules of what is right or wrong, states: 

We seek an understanding of ourselves so that we will know what we ought to do in any given situation. 

You will gradually replace the beliefs you were taught with the truths you discover. It is not whether you 

turn the other cheek that is important. Why you do or do not is most significant (p.37). 

These two points might appear diametrically opposed to one another but even here there is symbiosis to the 
one who looks more deeply into these statements. We are beginning to understand far more about what our Chinese 
students are thinking and how they process the information we provide them. 

The degree to which we either proceed toward cultural/intellectual emasculation or redefine what is “right 
or wrong” remains to be seen; indeed, our research must be conducted with the utmost sensitivity and tact; of this we 
are very well aware. 

The qualitative/subjective statements of our subjects portend that we may expect similar expressions from 
professors and teachers in both China and Taiwan when Stage 11 begins - a series of focus groups held in both 
locations to ascertain teacher/professor attitudes toward the new technology initiatives now being implemented. 
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Our intent is to conduct these focus groups via video, on location in Taiwan and China. While the opinions and 
attitudes of educational administrators are always germane to the success of any new initiative, our aim is NOT to 
include such individuals within the focus groups. In this way, we are concentrating our efforts on the new target 
audience, teachers, and professors, for they will carry the brunt of the day-to-day work. 

Our proposed solutions are numerous; some have been tried and proved helpful; a few have failed; others 
are being tested at this moment. Even more will emerge, we hope, because of the work done during Stage II. 
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Information Technology-driven Education in Japan: Problems and the 

Solutions 

Haruko Uchida 
University of Oklahoma 

During the 1990s, Japan was behind other industrialized countries in embracing the IT revolution. 
Focusing on the field of education, this paper describes policies and actions taken by the Japanese government in 
order to catch up with other countries. Then, the paper reviews impacts of these policies and actions on Japanese 
education including improvements and problems. Implications of these impacts are discussed based on the findings 
of the empirical research. 



Introduction of the Internet in Japanese Education 

While major Asian countries have formulated national plans and taken action to incorporate Information 
Technology (IT) into their societies since the early 1990s, Japan did not enact a national policy until 2000. Kimura 
(2002) contrasted Japan with three other Asian countries (Korea, Singapore, and Malaysia) in their national policy 
related to IT. Singapore established its National Computer Bureau (NCB) and started working on its Civil Service 
Computerization Program in 1981. Since then, Singapore’s government announced a series of national plans to 
promote the IT revolution (e.g., and Masterplan for IT in Education in 1997, One Network for Everyone in 1996, IT 
2000 plan in 1992, and National IT plan in 1986) and carried them out. Korea and Malaysia followed by 
introducing IT plans during the 1990s. 

National demand for the Internet 

Although the Japanese government established a headquarters for promoting advanced information 
communication society in 1994 and announced basic plans for its IT revolution in 1995, no serious action was taken 
until 2000. During the 1990s, the IT revolution had been treated merely as a part of plans that were to promote 
deregulation and technological innovation. However, the Japanese government changed its viewpoint and started 
considering the IT revolution as a way of changing not only its marketing systems but also Japanese society itself. 
That change might be a result of the following factors: (a) The United States’ economic prosperity, credited largely 
the growing information communication industry; and (b) Japan’s backwardness in embracing the IT revolution 
compared with other nations in the Asia-Pacific region as well as major industrial nations. In 2000, the government 
organized the “IT Strategy Headquarters” and established “e-Japan” programs that aimed to “make Japan the 
world’s most advanced IT nation within five years” (IT Strategy Headquarters, 2001; Ishikawa, Imamura, Taguchi, 

& Ueda, 2001). 

The “e-Japan” programs established the following four priority policy areas: (1) building an ultra high- 
speed Internet network and providing constant Internet access at the earliest date possible, (2) establishing rules on 
electronic commerce, (3) realizing an electronic government, and (4) nurturing high-quality human resources for the 
new era (IT Strategy Headquarters, 2001). 

The Japanese government’s policies and actions 

As shown in their goal to “make Japan the world’s most advanced IT nation within five years” (IT Strategy 
Headquarters, 2001), the Japanese government emphasized the urgent need to promote the IT revolution. The goal 
of the revolution was to create a “knowledge-emergent society” in which all Japanese people can use and access 
information technology (IT Strategy Headquarters, 2001). In order to accomplish this, the Japanese government 
further demonstrated the need for “nurturing high-quality human resources” and proposed the following three 
objectives: “(1) all citizens need to acquire IT knowledge and skills to enjoy its benefits, and people’s intellectual 
creativity and logical thinking power should be enhanced; (2) human resources who instruct IT to the public should 
be secured to improve people’s information literacy, and (3) technical experts, researchers and digital content 
creators should be fostered to explore the frontiers of IT” (IT Strategy Headquarters, 2001). 

Eurthermore, a series of actions that the Japanese government needed to take in order to achieve the 
objectives was listed. Eirst, in order to improve information literacy, the government addressed the need of 
“digitization of education” (IT Strategy Headquarters, 2001), that is, to enable IT-driven education by providing 
Internet access to all public elementary, junior, and senior high schools. In addition to enhancement of information 
literacy, the goals of IT-driven education included the following: to improve English proficiency since English was 
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the most important language in the Internet era; to promote skills in mathematics and science to foster the ability of 
thinking logically; and to cultivate ability of self-expression and creativity. Concurrently, interactions via the 
Internet with other schools both inside and outside Japan to foster collaboration with people who had different 
cultures and viewpoints were encouraged (IT Strategy Headquarters, 2001). 

Second, the government indicated a need to foster IT instructors by providing IT training for public school 
teachers. The goal was to enable all current teachers to instruct students using IT. In addition, a registration and 
dispatch system of IT instructors was introduced, so that persons with technology backgrounds in businesses and 
colleges could be utilized as IT instructors at schools (IT Strategy Headquarters, 2001). 

Outcomes 

Accomplishments reported by the Japanese government 

Since the establishment of e-Japan programs, there has been a significant improvement in the number of 
public schools that have Internet access: the rate of those schools rose from 57.4% in March, 2000, to nearly 100% 
in Aug, 2002, as shown in the report by the Ministry of Education (2002). In addition, a series of projects has been 
conducted since the 1990s with the Japanese government’s support to explore the advantages of using the Internet in 
classrooms. In these projects, nearly 100 schools were chosen and were provided with facilities for IT education. 
These schools demonstrated the effectiveness and potential of the Internet in education. Currently, the next project 
(“E square Project”) is underway providing support for schools that utilize the Internet on their own or through 
cooperation with other schools (Center of Educational Computing, 2002). 

Second, there has been an improvement in skills of IT instruction among public school teachers (e-Japan, 
2001). The government’s report shows that 781632 teachers (313271 in 2000) had participated in training related to 
IT education in 2001 (March, 2000 through March, 2001). Eurthermore, in March 2001, 79.7% of the teachers 
(66.1% in 2000) reported being capable of “using the computer” and 40.9% (31.8% in 2000) of the teachers reported 
being capable of “instructing with the computer” (The Ministry of Education, 2001). 

A gap between the government’s report and reality 

Ishikawa, Imamura, Taguchi, and Ueda (2001) identified several serious problems regarding the 
government-directed IT revolution. Eirst, a discontinuity existed between the government and Japanese citizens in 
regard to the goals of the IT revolution. Although it had been widely acknowledged among citizens that the 
government had been focused on IT, little had been revealed about the details of policies (i.e., objectives and 
detailed planned actions). Ishikawa et al. (2001) attributed this discontinuity to ineffective and insufficient 
government public relations activities and mass media’s indifference to the issue. 

Second, Ishikawa et al. (2001) described the results reported by the government as questionable, since 
some results were too optimistic to reflect reality. In particular, a report that showed nearly half of the Japanese 
population had been using the Internet (up from 21.4% in 1999) might have not been true. Thus, Ishikawa et al. 
(2001) indicated a need to re-examine the appropriateness of the methodology used for data collection. If the report 
did not accurately reflect reality, the goals set by the government were not appropriate since these goals were based 
on faulty research (Ishikawa et al., 2001). 

Einally, Ishikawa et al. (2001) expressed a doubt about the usefulness of the IT revolution itself, describing 
that there had been few examples that the IT revolution directed by the government changed the structure of the 
society as it intended. Although there was a flourish of IT related industries, which accompanied the endorsement 
of Clinton/Gore information superhighway in the United States, it might have been largely the result of the 
popularization of the Internet rather than the government’s support. Moreover, those IT related industries had 
declined since 2000. Therefore, Ishikawa et al. (2001) predicted that there would be criticism directed against the 
Japanese government, which has been investing large amount of national budget into the IT revolution. 

Based on observations at several elementary and junior high schools in Tokyo, Nakauchi (2001) listed a 
number of problems related to the use of IT at schools. One of the most serious problems was a lack of officially 
assigned system managers who were to protect a school’s on-line security. At most schools, teachers who had more 
experience in using the computer relative to others were assigned to take care of the school computers. Not all those 
teachers were aware of the importance of security. Even when they were, it was often difficult to obtain the 
cooperation of other teachers, some who believed that computers had nothing to do with them (Nakauchi, 2001). 

Second, Nakauchi (2001) pointed out a lack of experience and knowledge of using computers and the 
Internet among many teachers. Eor example, a teacher asked Nakauchi how to open “Yahoo!” on his personal 
computer which did not have an Internet connection; the same teacher also asked where he could purchase the 
“Yahoo!” application since his personal computer did not currently have it; some teachers deleted all the unused 
files including DLL and SYS files in order to make the computer work faster; and some teachers thought that they 
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should not use the Internet in a classroom unless they learn and understand everything about the computer. 
Nakauchi (2001) indicated the existence of a digital divide within a school: teachers who were capable of using the 
computer and the Internet utilized the computer in class as it was necessary, while those who did not have enough 
skills rarely used the computers for instruction. 

Moriyama (1996) stated that the encouragement (or enforcement) of connecting schools to the Internet 
might not have the desired impact on education unless we find an answer to the question of whether computers and 
the Internet are necessary in classroom or not. 

The next section reviews research conducted in the United States and in Japan to explore what potentials 
and advantages the Internet brings into instruction and what should be considered in order to make the best use of 
the Internet. 



What the Internet Brings Into a Class and How It Can Be Achieved 

Findings from research in the United States 

What effect does classroom use of the Internet have? Hemenway (2000) listed effects identified by high 
school teachers who used the Internet concurrent with classroom instruction at least once a week. First, the Internet 
impacted students’ motivation. Students were highly engaged, saw things in a new way, explored on their own, 
discussed ideas, and became more anxious to do projects. This was primarily the result of the availability of 
information outside the classroom. The Internet “makes many resources, including original source materials from 
all over the world, available to students, teachers, and school administrators” (Reid, 2002, p. 30-31). This 
availability affected the roles of students and teachers. Students played the role of active learners who seeked the 
necessary information on their own and exchange ideas with their peers, while teachers played the role of coaches 
guiding them toward how to find information. The availability and change in roles resulted in an improvement of 
the quality of the motivated students’ project (Hemenway, 2000). 

In addition, the Internet brought great advantages to students in rural areas. Without the Internet, their 
resources had been limited to textbooks and references in a small library. However, the students could obtain up-to- 
date worldwide information with the Internet in easy and quick way, like those who lived in urban areas 
(Hemenway, 2000). 

Findings from the “100-school Networking Project” in Japan 

The 100-school networking project began in 1996 in Japan, sponsored by MITI (Ministry of International 
Trade and Industry) and the Japanese Ministry of Education, and supported and managed by IPA (Information- 
technology Promotion Agency), CEC (Center for Educational Computing), and CII (Center for Information 
Infrastructure). The purpose of the project was to conduct joint research to demonstrate the effectiveness and 
potential of the Internet in the field of education (Center of Educational Computing (CEC), 2002). In the project, 
approximately 100 schools were selected from 1,543 applicants and developed activities using the Internet on their 
own or through cooperation with other schools (CEC, 2002). 

Akahori (1998) listed examples of the activities demonstrated by the participating schools including: 
collaborative learning in environmental education (e.g., an acid rain project in which 40 schools participated); report 
making by enlisting the aid of experts in social studies, in which volunteers answered students’ questions; weather 
study by connecting to the weather bureau and having students compare weather charts with pictures produced by 
weather satellites; cross-cultural education by connecting to overseas schools; posting web-based questionnaires on 
the peace problems and having students discuss responses to the questionnaires from all over the world; and 
problem- based learning by connecting to specialists (Akahori, 1998). 

Eurthermore, Akahori (1998) summarized impacts of the Internet use reported by the participating schools. 
Eirst, “the change of knowledge resources from a limited school space to expanded information resources 
worldwide” (Akahori, 1998, p. 1154) was listed. Traditionally, school resources had been limited to teachers’ 
lectures and textbooks that were examined by the Ministry of Education prior to their publication. The expansion of 
resources via the Internet impacted types of students’ knowledge from fixed to dynamic and alive (Akahori, 1998). 

Second, the Internet enabled collaborative learning in which students exchanged different perspectives, 
which promoted cross-cultural understanding. Prior to the use of the Internet, individual learning had been 
dominant, which had provided fewer opportunities for exchanging ideas and ways of thinking through interaction 
with other learners (Akahori, 1998). 

Third, Internet use changed the type of instruction and thus altered the roles of teachers and students. Prior 
to the use of the Internet, memory-based teaching had been the dominant style in which a teacher played the role of 
knowledge transmitter. The use of the Internet enabled a student-centered learning style in which students searched 
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and retrieved information and further constructed their own knowledge structures through the problem-solving 
activities. In these activities, the teacher became a facilitator for assisting students solving the problem (Akahori, 
1998). 

Finally, communication via the Internet highlighted the importance of reading and writing skills, while “old 
media,” such as face-to-face or telephone, focused on listening and talking skills. Several activities in the 100-school 
networking project reported students’ progress in reading and writing (Akahori, 1998). 

Effective use of the Internet 

As reviewed in the previous section, research has shown that the Internet has potential to improve 
instruction and to motivate students to become active learners. However, research also shows that the use of the 
Internet does not always impact instruction and learners as expected. For example, in Hemenway’s (2000) interview 
with high school teachers, one teacher reported few differences in the classroom since he had started using the 
Internet. He commented, “Training is limited, we need more on how to use the tools. I don’t have a sense of what 
is valuable, what I want them to produce, or how I would evaluate what I want them to do.” (Hemenway, 2000, p. 

1 18). Given the ambiguity of its utility and effectiveness, what should be considered in order to take advantage of 
the Internet use? 

Integration of technology to instruction 

Elsenberg (2001) argued that the goal of computer-based instruction was to lead students to become 
“computer literate” (p. 45), which did not simply mean a skill of operating a computer but addresses skills of using 
technology (not limited to a computer) flexibly and creatively. He esplained, “We want them [students] to be able to 
size up a task, recognize how technology might help them to fulfill the task, and then use the technology to do so.” 
(Elsenberg, 2001, p. 46). Elsenberg and Berkowitz developed a process model named “The Big6 approach” to teach 
information and technology skills (Elsenberg, 2001). 

The purpose of the model was to help people of all ages solve information problems. The model identified 
the following six stages that people were to go through when they sought or applied information to solve a problem 
or make a decision; task definition, information-seeking strategies, location and access, use of information, 
synthesis, and evaluation. Although these six strategies did not necessarily occur linearly, all stages were completed 
in almost all successful problem-solving situations. 

The model aimed at helping teachers who were to develop instruction in which students learn “integrated 
information and technology skills” (Elsenberg, 2001, p. 46) instead of learning “isolated computer skills” 

(Elsenberg, 2001, p. 46) that used to be a focus of a computer-based instruction. Integration referred to using 
technology in content areas so that the students were able to complete tasks in more effective and efficient ways. 
Eurthermore, students were expected to be able to apply technology-related skills to real situations and needs, that is, 
“to identify information problems and be able to locate, use, synthesize, and evaluate information in relation to those 
problems” (Elsenberg, 2001, p. 47). 

Skills of integrating the Internet to instruction 

Based on the stages of The Big6 approach, Schrock (2001) listed critical skills that teachers need to obtain 
in order to integrate the Internet in their instruction; searching for the target information, evaluating source (Web 
sites) of the information, and using models to incorporate the Internet use into instruction. 

The skill of searching included selecting the appropriate search method and locating information. In 
particular, teachers must be knowledgeable of two basic methods of finding information on the Internet; directories 
and search engines. Directories such as Yahoo! Were considered to be useful when teachers were new to the 
Internet or need to gather some ideas to decide what information was needed. On the other hand, search engines 
such as AltaVista or HotBot could be useful when teachers had a focused topic and needed to combine relevant 
terms. To locate the information needed, teachers should be able to do tightly structured advanced searches by 
limiting and broadening a search (Schrock, 2001). 

After locating the information, evaluation of the information and the source were necessary. Schrock 
(2001) presented criteria used in the evaluation; technical quality of the page and quality of the content. Technical 
quality included a consideration of the technical constraints at the school and at students’ homes (e.g., the speed at 
which a page loads onto the computer’s Web browser). Content criteria included the authenticity of the information, 
the authority of the author, the applicability of the information, and the presence of any bias. Furthermore, teachers 
needed to be aware of and to follow the copyright guidelines when using information from a Web site in a classroom 
lesson (Schrock, 2001). 
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When teachers obtain information from the Internet, they can create lessons that primarily use Internet 
resources and have students employ higher-order thinking strategies to complete the task. Schrock (2001) 
recommended Internet-based lesson model such as the WebQuest (developed in part by Bernie Dodge of San Diego 
State University). The WebQuest provided teachers with a workable model in which the Internet was incorporated 
into the curriculum. However, in order to use the WebQuest, teachers must acquire skills of searching and 
evaluating the target information on the Internet (Schrock, 2001). 

Need for Teacher Training 

The previous section shows that it is important for technology-based instruction to focus on “integration” 
rather than simply teaching the skill of operating a computer. In particular, in order to integrate the Internet to 
facilitate students’ meaningful learning in a content area, it is necessary for teachers to know how to search and 
locate the information needed, how to decide what information should be used in accordance with its quality and 
affordability, and how to incorporate the Internet use into an instruction. The challenge is to equip teachers with 
these skills. 

Petropoulos (2001) identified stages that were necessary for technology integration including budgeting 
and planning to get equipped with the required hardware, software and peripherals, making a plan to integrate 
technology with instruction, preparing teachers for the integration, guiding students in using technology to complete 
tasks, and looking forward to adopt new technology as needed. Petropoulos (2001) further argued that the most 
important step in the whole process was the third step (i.e. preparing teachers for the integration), which could be 
provided by an effective teacher training. 

Discussing what was necessary for effective teacher training, Petropoulos (2001) identified problems in the 
current teacher training sessions that were often ineffective. First, teachers often were rushed through “one shot” 
training sessions for introducing technology. Second, most of the training sessions focused on learning 
technological skills such as learning software applications instead of integration. Finally, these training sessions did 
not provide teachers with sufficient time for reflection and practice. As a solution, Petropoulos (2001) argued that 
teachers would need time to plan, practice, search, and collaborate to develop curriculum-based technology projects. 
Importantly, teachers should realize that they did not have to be experts in the field of technology. Instead, they 
needed skills and experiences to integrate technology into their instruction (Petropoulos, 2001). 

Currently, a number of training projects that focus on integration of technology are available, such as the 
West Virginia K-12 RuralNet project (Koul, Wiesenmayer, and Rubba, 2001) and the Educational Technology 
Professional Development Program (Ivers, 2001). In these programs, teachers’ perceived level of technology 
proficiency as well as their self-efficacy beliefs toward teaching with the Internet are measured prior to the training 
and are monitored during and after the training to evaluate the effects of training. 

Conclusions 

The previous section reviewed research and projects that have shown that the Internet has the potential to 
improve students’ motivation as active learners and their ability to complete tasks with greater quality in each 
content area. In addition, the 100-school networking project in Japan has revealed that the Internet enhances 
collaborative learning in a class, among classrooms and schools inside and outside of Japan, and between students 
and experts in the area. Furthermore, the findings in the project suggest that Internet use impacts types of instruction 
and learning, as shown in research in the United States. 

The Japanese government realized the urgent need to promote Information Technology (IT) including in 
the field of education and have worked on the policy, “e-Japan.” The goal was to create a “knowledge-emergent 
society” in which all citizens can use and access information technology. In order to accomplish the goal, the 
government planned to provide Internet access to all public schools to enable IT-driven education, which teaches 
information literacy as well as skills that were required in the “knowledge-emergent society” such as English 
proficiency, logical thinking skills, and creativity. Eurthermore, the government provided training sessions to 
current teachers, so that all current public school teachers would become able to instruct using IT (IT Strategy 
Headquarters, 2001). 

As a result, the government reported striking improvements in the number of schools that were equipped 
with Internet access and the number of teachers who were capable of “using the computer” and “instructing with the 
computer” (e-Japan, 2001; The Ministry of Education, 2001, 2002). The successful reports on the government- 
supported projects, including the 100-school networking project, highlighted the promotion of the IT revolution in 
Japanese education. 

However, a number of researchers and school staff members have questioned the accuracy of the 
government report. Eor example, Ishikawa et al. (2001) argued that the report was too optimistic to reflect reality. 
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Likewise, Nakauchi’s (2001) observation at schools indicated that environment and teachers were not ready to 
instruct using the Internet at many schools. 

What causes the gap between the Japanese government’s report and the reality of IT implementation? How 
should the government capture the reality in terms of readiness of teachers? And most importantly, what is 
necessary to be considered in order to improve Japanese IT education? 

The Japanese government reported an improvement of teachers’ IT using skills. The report was simply 
based on the increased number of teachers who answered “yes” to “being capable of using the computer” and “being 
capable of instructing with the computer”. As Ishikawa et al. (2001) indicated, the vagueness of the research was 
apparent in several areas. First, it was unknown whether the answer had been based on each teacher’s self-report or 
an observation by other teachers or a principal. Second, the statement “being capable of using the computers” did 
not address any particular skills that were required in teaching and completing school-related jobs. Likewise, the 
statement “being capable of instructing with the computer” was too vague to capture what exactly the teacher was 
able to do with the computer in order to promote instruction. Furthermore, the answers to the statements were 
categorical (i.e., yes or no) and did not refer to the degree of confidence. 

In order to identify how much effort and to what point it is necessary to prepare teachers for integrating IT 
to produce effects, scales that measure each teacher’s weakness and strength, as well as their beliefs about using IT 
in instruction are necessary. Importantly, the training sessions must evaluate an individual teacher’s skill in order to 
identify his or her need prior to the sessions and after the session to track the effects of the sessions (examples of 
these measurement used in the training were shown in the West Virginia K-12 RuralNet project (Koul et al., 2001) 
and the Educational Technology Professional Development Program (Ivers, 2001)). 

As reviewed, effective teacher training is critical for the successful integration of the technology. 

However, Misonou (2002) identified problems in IT training for teachers in Japan. He indicated that current training 
was mostly offered with a limited number of intensive courses, therefore, it would be difficult for many teachers to 
schedule for and to retain skills after the training. In addition, the training reflected little consideration of the needs 
and skills of each trainee. His suggestion should be applied to the government’s plan regarding the training, which 
emphasized the need to provide flexible and various types of training, in which each teacher can choose to take what 
she/he needs and at her or his convenience. 

In addition, Petropoulos’s (2001) suggestions are critical. That is, it was necessary to provide teachers with 
time to plan, practice, search, and collaborate to integrate technology to instruction. In order for Japanese instructors 
to integrate technology both effectively and efficiently, it is necessary that findings from the projects such as the 
100-school networking project be analyzed and shared by all teachers and schools. Currently, an effort to support 
schools on the Internet by providing schools with the findings of the projects and encouraging collaboration between 
schools has been made in the ongoing project (i.e., E square Project). However, in order to develop instructional 
designs that effectively integrate technology to instruction, collaboration should be expanded to include researchers 
as well as schools and teachers. 

Although the Japanese government includes improvement of a variety of skills such as logical thinking skill 
in its plan, actions were primarily focused on skills and knowledge to utilize the Internet. However, it is more 
important for classroom instruction to develop the strategies to improve learning using available technology than to 
teach how to use a specific technology. Since the Internet is a fast-moving technology, incorporation of the newer 
technology will be required as they become available and commonplace (Schrock, 2001). Research that develops 
instruction to effectively integrate ongoing technology to take full advantage of it on learning in a content area is 
necessary. 

Eurthermore, the formation of the community based on mutual understanding among the government, local 
governments, researchers, and members of schools (e.g., administrators, teachers, students, and their parents) is 
necessary. The community should be able to support meaningful learning by students with the Internet by providing 
teachers and schools with proper training and facilities. 
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Abstract 

A central issue in chemistry education is the relation between the macroscopic or real world and the 
molecular or nanoscopic world. New students could better understand chemistry and apply their chemistry 
understanding to solve problems if they were able to make deeper connections between these worlds. Animations 
can be used in chemical education so that students get a better knowledge of molecular processes by making better 
relations between the macroscopic and the nanoscopic world. 

Twenty 10'^ grade, pre-university students from four different schools were distributed into four five- 
student groups, students of each group attended the same school. Group 1 received instruction on paper. Group 2 
students were also provided with animations that showed nanoscopic processes. Group 3 received the same material 
as with Group 2, but the students were also required to complete assignments and perform a number of tasks for 
which they had to take a closer look at the animations. Students in Group 4 were required to do everything that 
Group 3 were required to do, and they were asked to make animations by themselves. The students were interviewed 
after instruction. 

Only two students (from Group 1 and 2) could give a complete, molecular explanation why ice floats on 
water when ice is melting. Six students (two from Group 1, three from Group 3 and one from Group 4) gave an 
explanation that was correct, but not complete. The other twelve students could not explain this phenomenon. Solid 
salt and distilled water are insulators, but a solution of salt in water conducts electricity. Students in Groups 3 and 4 
could explain this when they were performing the tasks about the animations. The Group 4 students made 
animations that were close to the scientific accepted models, which strongly indicate that creating animations gives 
a strong learning effect. Nevertheless, there was not a very clear distinction between the four groups when students 
were interviewed two weeks after instruction. 

The students came from four different schools, and had therefore different backgrounds. There will be a 
follow up investigation in order to get a more homogenous group, which will result in stronger conclusions. 

Introduction 

A computer animation is a series of rapidly changing pictures on the computer screen, which gives an 
illusion of motion (Large, 1996). There have to be at least fifteen pictures per second for a fluent and continuous 
motion. According to Mayer and Moreno (2002) an animation has three characteristics. It is a picture, it shows 
apparent motion, and it is simulated. This means that an animation consists of objects that are drawn or created with 
some other simulation method. A video shows motion of real objects. Similarly, an illustration is a static picture of a 
drawn or simulated object, and a photo is a static picture of a real object. Computer animations can be used in 
instruction programs on CD-ROM or on the Internet. Movement of so-called gif-animations cannot be stopped; the 
animations cycle in a loop. Other animations play via a plug-in like QuickTime or Flash*. QuickTime animations 
can be stopped, so certain details can be viewed more closely. It is possible to play animations in programs such as 
QuickTime or Windows Media Player, and to copy frames of the movie. This allows students to interact with the 
animation. 

Animations can serve different roles in instruction. Weiss, Knowlton and Morrison (2000) mention 
decoration, gaining attention, motivation, extra information and clarification of complex knowledge or complex 
phenomena as potential roles of animation. Gaining attention is an important function, in which there must be 
overlap with the content of the accompanying text. If there is no overlap, the animation can distract from the 
instruction. Large (1996) argues that animations add to written information, but cannot replace it. Motion is the 
special quality of animations and therefore animations can promote learning of dynamic processes (Large, 1996). 
Since chemical processes at the molecular level are dynamic, impossible to see, and typically quite hard to imagine. 
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animation could be a powerful tool in chemistry education. Atoms, molecules and ions are not static, but vibrate, 
move, collide and interact with each other. These dynamic processes are better represented in an animation than in 
static pictures. Molecular-level animations have been proposed as a way to support student understanding in 
chemistry (Burke, Greenbowee & Windschitl, 1998). The rationale is that animations make visible otherwise 
abstract chemical concepts, especially those related to the particulate nature of matter. 

Chemistry explains many processes from the real world with the abstract concepts of atoms, molecules and 
ions. The size of atoms, molecules and ions is typically several nanometers, so the world of these particles is 
sometimes called the nanoscopic world (Schank & Kozma, 2002; Vermaat, 2002). Other names are the molecular 
world, submicroscopic world (Ebenezer, 2002), or particle world (Bunce & Gabel, 2002). The name microscopic 
world (Sanger & Greenbowee, 2000; Wu, Krajcik & Soloway, 2001) suggests that the particles can be seen through 
a microscope. This is not the case, so the terms nanoscopic or submicroscopic are preferred. Because atoms and 
molecules cannot be perceived directly, chemists work with models or representations, constructions on paper, 
computer-based animations, or physical constructed models that have some properties of the real object. A ‘good’ 
model shows the appropriate properties of the associated object (Dieks, 1999). A goal of instruction is that students 
develop mental models that look like conceptual models that are accepted by the scientific community, and that 
students fit these models into a structure of related knowledge (Greca & Moreira, 2000; Seel, 2003). Mental and 
conceptual models are contrasted in Table 1. 



Table \.The differences between mental and conceptual models (after Greca & Moreira, 2000). 



a mental model 


a conceptual model 


is a personal, internal representation, used by 
learners (mental models can be external if 
learners make sketches). 


is a public, external representation, created by 
researchers, teachers, experts, etc. 


explains and makes predictions about an 
associated system. 


facilitates the comprehension or the teaching of 
systems or states of affairs in the world. 


has to be functional to the person who 
constructs it. 


has to be useful to the (scientific) community. 


is incomplete, not exact, changeable 


is more complete, more exact, less changeable 


grows and becomes better if new knowledge is 
acquired. 


is more or less full-grown. 


can be quite different from the real object, 
phenomenon, or situation it should represent. 


is a simplified representation of a real object, 
phenomenon, or situation. 



Chemists use a range of representations to understand scientific phenomena. They switch between different 
representations and use them in a combined fashion to solve scientific problems, to predict certain phenomena, and 
to communicate with other chemists. However, students typically lack both the basic knowledge and the skills to 
work with different representations (Kozma & Russell, 1997). Because of limitations of representational knowledge 
and the skills to use these representations as objects of thought, students often do not understand scientific symbols 
(Kozma, 2000; Seel & Winn, 1997). They could better understand chemistry and apply their chemistry 
understanding to solve problems if they were able to make deeper connections between reality, the molecular world 
and the world of chemical formulas and equations (Herron, 1996). 

Students should also be prompted to make relations between the three chemical worlds. Researchers 
hypothesize that the difficulties students face in relating elements among the three levels are at the root of their 
conceptual problems (Gabel, 1998; Nakhleh, 2002). Not only should students develop mental models that are 
scientifically accepted, but also they should use these models to explain and/or predict macroscopic phenomena. 
Furthermore they should be able to translate the macroscopic and nanoscopic world into the symbolic world and 
vice versa. Chemists communicate by chemical symbols, formulas and equations of the symbolic world. Instruction 
and assessment should explicitly address the relations between the three chemical worlds. 
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A process that is explained in the nanoscopic world is the floating of ice on water, a concrete phenomenon 
that is known to most students. In nearly all substances the solid phase is denser than the liquid phase, and when 
both phases are mixed, the solid phase will sink. This can easily be seen if one puts a bottle of olive oil in a freezer; 
as the oil cools and solidifies, it forms a solid on the bottom. In the Netherlands, 9* grade students are taught a 
model for solid, liquid and gaseous phases (Figure 1), which explain this phenomenon (Camps, Pieren, Scheffers- 
Sap, Scholte & Vroemen, 2002; De Valk & Ousen, 2003; Hogenbirk, Jager, Kabel-van den Brand & Walstra, 1999). 
However, this model does not explain why ice floats on water. According to this model, a solid will be denser than a 
fluid, because in a certain volume of solid there are more molecules than in the same volume of liquid. In order to 
explain the lower density of ice, students have to change their mental model of ice and water. 
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Figure 1 .Models of a solid, liquid and gas, as taught in Dutch schoolbooks. Molecules in solids are immobile, in 
liquids there is motion, and molecules in gases don’t attract each other at all. 

Another familiar phenomenon is the electrical conductivity of a solution of salt in water. Pure, distilled 
water and solid salt (sodium chloride, NaCl) are insulators for an electrical current, while a solution of salt is a 
conductor. Due to the curriculum in the Netherlands, 10* grade students think that all substances consist of 
molecules or atoms. They know that an electric current is transfer of charge. To explain the conductivity of a 
solution of salt in water, students have to change their mental model of the building of substances, and to accept that 
there are substances that consist of charged particles. 

Both the melting of ice and the dissolving of salt in water, which results in a conductor, are complex 
dynamic processes. Animation should therefore be a useful tool of instruction for these processes. Just showing 
animations in order to change the mental models of the students will probably not be enough. In an animation there 
are a lot of changes simultaneously, and it is difficult for students to recognize the conceptually important features of 
the animation (Schank & Kozma, 2002). Furthermore, students tend to hold to their old models, and disregard 
information that contradicts this model (Herron, 1996). 

To change the mental models of the students, they should work actively with animations and recognize and 
comprehend the important features of the process. This can be done by completing assignments and performing 
tasks in which students have to take a close look at the animations. If students have to create animations by 
themselves, they will have to imagine carefully what the nanoscopic processes are. Students in earth science who 
created diagrams of what they learned about plate tectonics outperformed students who wrote summaries of their 
learning (Gobert & Clement, 1999). Students who make animations probably will develop strong mental models, 
which will be more in accordance with scientifically accepted models. 

A tool that is used for students to create animations is ChemSense, a NSF-funded project by the Stanford 
Research Institute at Menlo Park, CA, U.S.A. (Schank & Kozma, 2002). The ChemSense team has designed 
software and activities to help students to connect observable, macroscopic phenomena with nanoscopic 
representations, and explain these phenomena in terms of the underlying, nanoscopic mechanisms. The ChemSense 
Studio supports the viewing, editing, and sharing of various representations, including animations. Students and 
instructors can comment on representations that are made by other students. Research by Schank and Kozma (2002) 
suggests that the use of ChemSense facilitates representational ability and understanding of nanoscopic mechanisms. 
The use of ChemSense requires students to think carefully through more specific aspects of chemical phenomena to 
which they might not otherwise attend. Students arrived at a shared understanding of the chemical content through 
their planning and discussion of animations. 
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The purpose of this study is to investigate how animations can be used in chemical education so that 
students gain a better understanding of molecular processes and improve their abilities to relate the nanoscopic and 
the macroscopic world. 



Method 

Participants 

Twenty 10* grade, pre-university students from four different schools participated in this study. These 
twenty students were distributed into four five-students groups; students of each group attended the same school. In 
this way there was no contact between students of different groups. 

Group 1 received instruction on paper (see Materials). Their teacher had already introduced the processes 
of the melting of ice and the conductivity of a salt solution during regular lessons, and the students had completed a 
test about these topics. 

Students in Group 2 received instruction on paper too, but they were also provided with animations^ (see 
Materials). There were no special assignments or tasks belonging to these animations, the students were just asked 
to take a look at them. During this investigation, these students also received instruction in their regular lessons by 
their own teacher about the processes. 

The students in Group 3 received the same instruction materials as those of Group 2, but they were also 
required to complete assignments and perform a number of tasks (see Materials) for which they had to take a closer 
look at the animations. Group 3 students, just as those of Group 1, had received instruction about the processes by 
their own teacher during regular lessons before this research. 

Group 4 students were required to do the same assignments and tasks as the Group 3 students, but they also 
made animations by themselves. Due to time restrictions. Group 4 students made only animations of the dissolving 
of salt. The students in this group were instructed by their own teacher about the processes of melting of ice and 
conductivity of a solution of salt after this research. That is, unlike Groups 1-3, Group 4 students had no prior or 
concurrent instruction on the relevant topics, which (in retrospect) makes it difficult to compare their results to the 
other groups. Some interesting results are discussed nonetheless, and this methodological limitation (which was due 
to logistical constraints) is discussed below and will be remedied in a future study. 

Materials 

The instruction on paper for the students consisted of copies from the textbook that is in use at their 
school. The students were asked to study these papers by themselves. For the students of Group 1, 3 and 4 these 
were copies from the textbook Chemie (Pieren, Scheffers-Sap, Scholte, Vroemen & Davids, 1999); for Group 2 
students the copies came from the textbook Curie (Van Antwerpen, Bouma, Le Fevre, Van Schravendijk, Schouten, 
Van Steeg & Termaat, 1998). 

The animations shown to the students of Group 2, 3 and 4 were made by Roy Tasker of the University of 
Western Sydney, Australia (Tasker, Chia, Bucat & Sleet, 1996). The first animation depicted the melting of ice at 
the molecular level. Frames from these animations can be seen in Figure 2. The second animation depicted the 
dissolving of table salt (sodium chloride) in water. Group 3 and 4 students also heard a recorded voice, in English, 
that explained the process. The students were given a Dutch translation of this talk. In addition, there were 
animations of a hydrogenated chloride ion and of a hydrogenated sodium ion. Frames of these animations can be 
seen in Figure 3. 




Figure 2. Frames from the animation 'Ice melting' 
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Figure 3. Frames from the animation 'Dissolving of salt'. 

Students of Group 3 and 4 had to complete the assignments shown in Table 2 while working with the animation. 
Group 4 students used the ChemSense Studio^ for making animations by themselves. 
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Table 2. 



Assignments accompanying the animations 



Assignments for the animation ‘Ice melting’ 



• Take a look at the animation and describe what you see. 

• After how many seconds the ice starts to melt? How can you see this when looking at the water 
molecules? 

• Where is more space between the water molecules, in ice or in liquid water? 

• What is denser, ice or liquid water? Explain. 

• Are there more watermolecules in a cm^ ice, or in a cm^ liquid water? Why? 



Assignments for the animation ‘Dissolving of salt’ 



• Are the gray balls sodium ions (Na^ ions) or chloride ions (Cl“ ions)? Explain. 

• Stop the animation at the moment the first (green) ion will be surrounded by water molecules. 

• What is the charge of this ion? 

• Which side of the water molecules is pointed to this ion, the side with the hydrogen atoms, or the 
side with the oxygen atom? 

• Stop the animation at the moment the second (gray) ion will be surrounded by water molecules. 

• What is the charge of this ion? 

• Which side of the water molecules is pointed to this ion, the side with the hydrogen atoms, or the 
side with the oxygen atom? 

• A water molecule is a dipole molecule, it contains a 6+ and a 5- side. 

• Explain that the hydrogen side of the water molecules is pointed to the chloride ion. 

• Explain that the oxygen side of the water molecules is pointed to the chloride ion. 



Instruments 

All the students were interviewed about two weeks after the instruction. These interviews were semi- 
structured, the interviewer used the same open-ended questions for each student (Table 3). The interviewer asked the 
students to draw nanoscopic processes that show what happens if ice is melting and salt is dissolving in water in 
order to explain macroscopic processes. 

Table 3 The interview questions 



Questions for the phenomenon melting of ice 

• What do you observe in the real world if ice is melting? 

• Does ice float on water? 

• What has the greatest density, ice or water? 

• Can you give a molecular explanation (an explanation in which you use the word molecules) for the 
phenomenon that ice floats on water? Make a sketch that shows what happens to the water 
molecules if ice is melting, or what the difference is between water molecules in ice and in liquid 
water. 



Questions for the phenomenon conductivity of solution of salt 

• Can you describe what happens to the water molecules and the particles of which salt consists if 
salt is dissolved in water? Make drawings that show what happens. 

• Can you explain why a solutions of salt does conduct a current and solid salt does not? 
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Procedure 

The interviews took place about two weeks after the students had received the instruction. 



Analysis 

All the interviews were taped and the drawings were scanned. The results of all interviews were written 
down. After that, the explanations of the students were classified and analysed. Special attention was given to the 
nanoscopic explanation that students gave for the macroscopic processes. 



Results 

Melting of ice 

In response to the question “What do you observe in the real world if ice is melting?” all twenty students 
eventually mentioned the change from a solid state to a liquid state. Eight students in their description started to talk 
about molecules. After the interviewer asked if they could see molecules, these students gave a macroscopic 
description. 

All students knew that ice floats on water and does not sink to the bottom if ice is melting. Eighteen 
students said that ice is less dense than water (e.g., one said, “Water is of course denser than ice.”). Two students, 
both from Group 1, thought that ice is denser than water. 

The explanation of the phenomenon that ice floats on water relates to the unique arrangement of the 
molecules in ice and in liquid water. In ice the water molecules are held together by hydrogen bonds in a hexagonal, 
relatively open structure as can be seen in the left part of Eigure 2. This orderly arrangement collapses during 
melting, and the molecules pack less uniformly but more densely (Jones & Atkins, 2000). During instruction, the 
students of both Group 3 and 4 formulated this explanation. However, two weeks later, no students from Group 3 or 
4 were still able to give a complete explanation (Table 4). Six students were able to give an incomplete explanation, 
that there is more space between the molecules in ice than there is in liquid water. Half of the students held on to 
their old, limited model of solids and liquids shown in Eigure 1 . Two students invented whole new explanations 
involving ‘surface tension’ and ‘loss or energy’. The last student could not explain what was meant by this concept. 



Table 4 Students' explanations why ice floats on water, two weeks after the intervention 



Group 


complete explanation 


in ice more space 
between molecules 
than in water 


alternative 

explanation; 

‘old’ model of solids 
and liquids 


other alternative 
explanation 


1 


1 


2 


1 


1 (surface tension) 


2 


1 




3 


1 (loss of energy) 


3 




3 


2 


4 




1 


4 


totals 


2 


6 


10 


2 



In sum, after two weeks, half of the subjects returned to the old model of solids and liquids, and only two students 
(in Groups 1 and 2) were able to give a complete, accurate explanation. Active use of animations did not lead to 
better explanations by Group 4 students, but this outcome could also be explained by the fact that Group 4 students 
had not received prior instruction on the topic. What is remarkable is how many students returned to the old model 
despite instruction with or without interaction with animations, illustrating how students’ difficulties in 
understanding chemistry concepts often persist despite instruction (Gabel, 1998). 

Dissolving of salt 

All students knew that a solution of salt conducts an electrical current, but neither solid salt nor pure water does. The 
explanation for this phenomenon is that in solid salt, all the ions (charged particles) are held together by ionic 
bonding, so the ions cannot move. During dissolving, the ions are pulled out the ionic lattice by the water molecules. 
The water molecules surround the ions. The slightly positively charged hydrogen atoms in the water molecule are 
attracted to the negative chlorine ions, while the slightly negative oxygen atom of the water molecule is attracted to 
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the positive sodium ions (Jones and Atkins, 2000). The sodium ions are smaller than the chlorine ions (ionic radii 98 
pm and 181 pm respectively). In a solution of salt, the charged particles can move, which means that the solution 
can conduct a current. The students of Group 4 made animations in which they showed that they understood this 
process (Figure 4). 




Figure 4 Frames from an animation made by a student of Group 4. 



The results of the interviews are listed in Table 5. A complete explanation includes the following items; 
in solid salts the ions are immobile; 
in a solution the ions can move; 

the sodium ions are positively charged and smaller than the negatively charged chlorine ions; 
the ions in a solution are hydrated (= surrounded by water molecules); 

the oxygen atoms of the water molecules are near the sodium ions, the hydrogen atoms of the water molecules are 
near at the chlorine ions. 

In an incomplete explanation, students state that in solid salts the ions are immobile, and in a solution the ions can 
move, but they; 

have inverted the charge or the size of the ions, and/or 
drawn ions of the same size, drawn a wrong hydratation, and/or 
used the formula HO 2 for water molecules, and/or 
reversed the charges of the water molecule. 

An alternative explanation lacks the immobility of ions in solid salt and the mobility of ions in a solution, which are 
the key features of this phenomenon. 



Table 5 Students’ explanations of the phenomenon that a solution of salt conducts a electrical current, two weeks 
after the intervention. 



Group 


complete explanation 


incomplete explanation 
(size or charge of ions, 
hydratation) 


alternative explanation 


1 




4 


1 (water is the conductor) 


2 




3 


2 (hydrogen bonds; no 
explanation) 


3 




4 


1 (particles vibrate) 


4 


1 


3 


1 (electrons are responsible) 
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Figure 5 shows two sketches made by students during their explanations. The sketch in the upper part of Figure 5 
was made by a student who gave a complete explanation. This student said that the Na particles are positively 
charged sodium ions and that the Cl particles are negatively charged chlorine ions. The circle with the “3” in it 
represents an water molecule, which was sketched as in the upper right part of Figure 5. The sketch in the lower part 
of Figure 5 shows an incomplete explanation. Both charge and size of the sodium and chlorine ions are inverted, as 
is the charge within the water molecules. 




Figure 5 Sketches made by students during their explanation that solid salt does not conduct an electrical current, 
but a solution of salt in water does. 

Discussion 

It is remarkable that eight out of twenty students started to talk about water molecules when asked: “What 
do you see if ice is melting?” They knew this research was about the way chemistry is taught, and they probably 
connect the chemistry with the concept of molecules. However, this does not mean that these students actually link 
the molecular world to the real world. It is possible that the concept of molecules is an abstract concept, which has 
little meaning to them. 

Half of the students held on to their old model of liquids and solids. This model is insufficient to explain 
the phenomenon that ice floats on water, and these students were not able to give a good explanation of these 
phenomena. In an informal conversation, a teacher mentioned that the model of Figure 1 has been ‘drilled’ in 9* 
grade students, and it may therefore be hard to change their mental model. 

From the data in this research, it is not clear that an active look at the animated model of water molecules in 
melting ice promotes the understanding of this process. Three of the five Group 3 students gave an incomplete 
explanation (in ice more space between the molecules than in water. The two students who gave a complete 
explanation came from Groups 1 and 2. However, the students of Group 1 and 3 had already received instruction 
about this process before this investigation, so for them the instruction during this research is a repetition. A higher 
proportion of correct explanations from students in Groups 1 and 3 should therefore be expected based on their prior 
knowledge. 

The animations made by Group 4 students are close to the scientifically accepted models. The students at 
once noticed some restrictions of ChemSense. They wanted to use the ionic radius for the sodium and chlorine ions, 
and the Van der Waals radii for the atoms in a water molecule. Furthermore, they expected green chloride atoms or 
chlorine ions instead of the blue ones they had to use in ChemSense. (In the time since this study was conducted, 
both ionic radii and more standard colors for atoms have been added to ChemSense). Even with the restrictions at 
the time of the study, the animations are quite detailed and accurate (Figure 4). All the significant features of the 
dissolving of salt are incorporated in the animation: in solid salts the ions are immobile, in a solution the ions can 
move, the ions in a solution are hydrated, the oxygen site of the water molecules is pointed at the sodium ions, and 
the hydrogen site of the water molecules is pointed at the chlorine ions. While they were making the animations, the 
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students mentioned that the sodium ions should be smaller than the chlorine ions and that the sodium ions have a 
positive and the chlorine ions a negative charge. So their mental model had changed to the conceptual model. 

The students of Group 1 and 3 had already received instruction about the process of the dissolving of salt, 
and the students of Group 2 were instructed by their own teacher during this research. The Group 4 students were 
instructed by their own teacher after this research. This difference in background knowledge may explain why 
Group 4 gave less accurate explanations during the interviews. Still, their explanations were as good as those from 
the other three groups, and the only complete explanation of the dissolving of salt was even given by a student in 
Group 4. This suggests that animation construction can be a powerful learning tool. 

During the instruction, the students of Group 3 and 4 made clear that they understood both processes 
(melting of ice, a solution of salt conducts a current). Two weeks later they seem to have forgotten certain features 
of the processes. This resulted in incomplete explanations for both processes and alternative explanations for the 
conductivity of a solution of salt (particles vibrate, electrons are responsible; Table 4). 

The students had a mental image of the floating of water on ice, and they knew that a solution of salt 
conducts a current, but they had not seen the processes itself at the start of the instruction. In order to enforce the 
link between the macroscopic and the nanoscopic world, the students should /irsf see the processes and then get 
instruction in which molecules and ions provide an explanation. 

For the first process, they could see a large beaker filled with water and ice. The instructor could point out 
the fact that ice floats on water. Then, the students should reason that liquid water is denser than ice, which means 
that there are more water molecules in a mL of liquid water than in a mL of ice. After that, students could be 
confronted with the model of Figure 1, and discuss whether or not this models is sufficient to explain the floating of 
ice on water. For the second process, the students could observe that neither pure water (distilled water) nor solid 
salt conducts a current, but that a solution of salt in water does. 

A weak point of this study that the four groups of students came from four different schools and had 
different backgrounds. Ideally, it would be better if all the students were in the same classroom with the same 
chemistry teacher, but logistically, is difficult manage four separate groups in one classroom and to keep the groups 
separated. This can only be done if the students are interviewed directly after the instruction, otherwise they could 
talk to each other or observe each others’ work, which would contaminate the measurement. 

A follow up study with four groups from the same school will follow soon to find out if constructing 
animations give better learning results than only looking at animations and completing assignments about them. The 
animations of Tasker (Tasker, Chia, Bucat & Sleet, 1996) are better looking and more professional than students’ 
animations made with ChemSense Studio, but there are strong indications that constructing animations gives a better 
learning effect than just examining them. 
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Abstract 

A high school science teacher reported that the students have motivation and learning problems to 
understand the concept of fossilization. Working with the science teacher over eighteen months, Fossilization Web- 
Based Learning Environment (Web-LE, http://iris.nyit.edu/~skwang/fossil) was designed and developed by a group 
of students in the Department of Instructional Technology at University of Georgia to investigate the impact of the 
this tool on student motivation and learning achievement. The results showed that this Web-LE provided an 
appropriate tool for the teacher to teach the unit of fossilization, enabled the teacher to design specific assessment 
strategies, and improved student motivation. The teacher will use the tool in the future class since Fossilization 
Web-LE is customized for him and it is proved useful in his situation-specific context. 

Introduction 

The primary setting of this developmental research was a local private school located in a small city in 
northeast Georgia. This school is equipped with well-organized computer technology including computer labs, 
laptop, and wireless network. Students and teachers are used to the computer environment and equipped with 
relatively higher computer literacy. Some teachers in this school looked for strategies of integrating technology into 
their classroom. One tenth grade science teacher reported that he needed assistance to develop interactive 
instructional materials in order to solve his instructional problems. The team from the Department of Instructional 
Technology at University of Georgia analyzed his problems and decided to work with him. Twenty two months 
were invested on this project to conduct a developmental research and design Fossilization Web-LE. Fossilization 
Web-LE was designed to utilize the unique features and elements of the World Wide Web (WWW) to enhance and 
sustain the motivation of learners in the context of secondary science education. The project falls within the category 
of developmental research goals described by Reeves (2000). The purpose of developmental research is twofold: to 
focus on developing approaches to improve situation-specific teaching and learning problems and to generate 
methodological directions for design and evaluation for future development and implementation efforts. 

Instructional Problems 

The science teacher reported that the students have problems to understand the concept of fossil formation. 
Eossilization is a result of complex combination of an organism, ecological condition and physical burial. The fossil 
would be formed only under the correct situation. Students have to understand that how and why ecological 
conditions and physical burials influence fossilization. The teacher couldn’t find accurate or useful materials to help 
students visualize the process of fossilization. Before the development of this tool, the teacher demonstrated 
volcanic films or pictures to help students to have a big picture of the fossilization conditions. He also asked 
students to pick up one condition and represent it by drawing. However, students could not understand the overall 
difference among the potential conditions of developing fossils. He needed moving illustrations describing different 
conditions of fossilization but he could not find appropriate materials to do so. The other problem was that students 
had relatively low motivation when learning the unit of fossilization. The science teacher needed a tool that is 
realistic and accurate to describe the process of fossil formation and provide opportunities for learners to consider 
the potential combinations of different decision. The learner will be able to identify situations that will cause 
fossilization. 

Instructional Design Strategies 

Several strategies for increasing students’ intrinsic motivation will be involved in the Web-LE. First, 

Lepper and Hodell’s (1989) four characteristics (challenge, curiosity, control, and fantasy) of tasks that promote 
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individual intrinsic motivation were integrated into the instructional design. Second, raises and sustains learners’ 
motivation by integrating multimedia objects into the learning context. Learning with multimedia provides an 
effective alternative instructional strategy (Mathewson, 1999). From the motivational perspective, using medium to 
assist instruction indeed enhance learners’ interest (Freeley, 1982; Kramarski & Feldman, 2000). The multimedia 
objects were employed to represent the fossil concepts to enhance intrinsic motivation. Third, the teacher designed a 
task requiring higher order thinking skills for the unit of fossilization and encourages students to explore all 
scenarios to generate the conclusions instead of learning by rote. 

Overviews of the System 

After completing learning the concept of fossilization with this cognitive tool, students will be able to 
identify conditions necessary for fossilization and construct possible scenarios for fossil formation by manipulating 
the variables in the simulated processes. Fossilization Web-LE includes eighteen different scenarios to describe 
various conditions of fossilization and provides students the best way to observe most potential conditions to 
develop fossils. This is a three-day learning session. The teacher distributed the assignment in the first day and 
students had to solve problems within three days by exploring the scenarios provided the tool. These problems 
required critical thinking abilities such as comparing the similarities and differences and infer the conclusion. 

When entering the Web-LE, learners will see the instructional page to explain the learning objectives and 
goals of the fossil unit as well as how to use the tool under the “manual” section .Eigure 1 is the flow chart of 
fossilization cognitive tool created in Elash. After reading the task of exploring this interactive learning model, 
students will be prompted by the main interface of this tool. They have to follow the order of selecting an organism 
(first), an ecological condition (second), and then a physical burial (third). The combination of three decisions will 
decide whether the fossil will be developed and the simulation of fossilization begins after the combination has been 
made. The dinosaur is the only available organism in this version. Some abstruse terms regarding fossilization were 
identified by the teacher and illustrated by animations and descriptions in the online glossary. Students can assess to 
the online glossary to look up detailed information if they have any questions about the scientific terms. 

The system will record paths that learners just made and help learners to identify the learning progress. 

The complete eighteen possible combinations including: 

Organism: dinosaur 

Ecological status (3 conditions): temperate rainforest, tropical rainforest and tropical mountains 
Physical burial (6 conditions): weathering, ashfall, lava flow, pyroclastic, swamp, and flood 




Eigure 1 . Elow chart of fossilization cognitive tool 



Media development 

The primary audience is high school students who are appeal to realistic graphics and animations, and their 
learning style is more visual oriented. The science teacher also requested for the realistic representation of the 
fossilization processes. Several endeavors were made to achieve the requirements. Referring to the accurate images, 
the dinosaur was developed with an accurate proportion and color. 3D software (Corel Bryce) was used to develop 
the realistic landscapes and objects including ancient plants, volcanoes, lakes and fossils in the fossilization 
animations. Elash MX (Macromedia) was the primary authoring tool because of its ability of integrating multimedia 
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and the capability of optimizing media to enhance the speed of transmission on the Internet. The team met the 
teacher regularly and worked closely with him to conduct formative evaluation to ensure the graphical objects, 
descriptions of scenarios, information organized were accurate and realistic to the target learners. 

Results 

The teacher and twenty-seven tenth-grade students participated in this study, spending three class sessions 
using Fossilization Web-LE to finish cognitive tasks assigned by the teacher. Motivation questionnaire data were 
collected at the end of the third session allowing students to evaluate the tool and examine their attitudes toward 
using it. The student grades were also collected and analyzed to determine their learning performance. Observation, 
teacher interview, and twelve focus student interviews were collected during the sessions and after the three-day 
study was over. 

The motivation questionnaire, observational protocol, interview questions were designed based on 
Newmann’s identification (1992, p.l3) of levels of engagement: (1) observable behavioral responses, (2) covert 
cognitive responses activated during learning, and (3) interest. The researcher carried out statistical analyses and 
correlation analyses to examine factors that might have influenced student motivation and learning performance. 
Newmann’s levels of engagement helped the researcher scrutinize indicators of motivation that surfaced during the 
learning process. Student grades, student interviews, and the teacher interview were used to triangulate the data 
collected from the motivation questionnaire. The results showed that students reaped moderate learning and strong 
motivational benefit through using Fossilization Web-LE. Several minor interface design flaws were found during 
the implementation. These problems have since been fixed so that the teacher can use the revised program with 
future classes. 

Lepper and Hodell’s (1989) four factors for improving motivation were applied to the design of 
Fossilization Web-LE, and the tool had positive impact on student motivation as a result. Most students admitted 
that compared to other units, motivation toward learning fossilization was relatively lower without using the 
software; such statements reflected the teacher’s statement of his instructional problems. Results collected from the 
motivation questionnaire and student interviews revealed an important message: student interest toward learning the 
topic of fossilization was higher when learning with Fossilization Web-LE. The results of implementation provided 
significant evidences that the tool solved the teacher problems. It enabled the teacher design various tasks that he 
could not do before using the tool and improved student motivation of learning the unit of fossilization. 
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Abstract 

Designers utilize design languages while they design. Notation systems, an embedded element of a design 
language, capture abstract ideas to create communicable designs. Ai afield matures, notations interact with design 
languages to improve both language and notations. This is important in the maturation of a technological field. The 
field of Instructional Technology can benefit from notation systems to record precise and minute details of designs 
in order to ensure correct duplication, execution and communication. It appears that a similar notation convention 
can emerge for instructional designers, but it may be many years or even decades before this happens. This paper 
illustrates a number of dimensions that can help designers form their design languages and select the 
characteristics of their notation systems. 

"As the musician needs to record the precise and minute details of his composition to insure 
correct performance of his score, so the [instructional designer] needs a notation capable of equal 

accuracy" —George Balanchine 

Overview 

Designers utilize design languages while they design. Notation systems, an embedded element of a design 
language, capture abstract ideas to create a communicable design. As a field matures, notation systems interact with 
design languages to improve both language and notations. This is important in the maturation of a technological 
field. 

The field of Instructional Technology can benefit from notation systems to record precise and minute 
details of designs in order to ensure correct duplication, execution and communication. The need of a recording 
system in dance served as an impetus for a dance notation system; a similar need in instructional technology is no 
less important or critical. A case study of dance notation to derive important principles for instructional technology 
design notation systems is used. 



Dance Notation 

A dance notation system is analogous to a musical notation system; it records the entire character of the 
score including information for musical counts, measures, rests, body placement on the stage, head direction, feet 
placement, arm movements and placement, action, interaction between multiple dancers, time signature, lighting, 
stage design, props, and many other aspects of the production. “It is an agreed-upon notation for capturing abstract 
[dance] designs in graphical, textual, and symbolic representation (Gibbons, work in progress).” 

Dancers are expected to memorize movements of ballets they perform, requiring continual classroom and 
“take-home” practice. To facilitate take-home practice, study guides are written to remind dancers of their 
movements in the ballet. These study guides are written using a notation system that identifies the body position, 
movement timing, and other important information already described. This case study examines two dance notation 
systems: (1) Labanotation, a geometric dance notation system used for centuries, and (2) the Sutton Dance Writing 
Method ®, which has used iconic caricatures to record dance movements since 1973. These notation systems also 
allow for the composition of new ballets, making them an invaluable design tool for choreographers. Without 
notation systems like these, dance designs would be limited by memory and hard to transmit across time. 

There are a great many aspects of a ballet production that must be weaved together to create a wonderful, 
meticulously crafted dance production. The various layers of dance design call for artisans experienced with the 
different layers of dance. For instance, people study costume, set, and lighting design for many years, both in school 
and in apprenticeships before becoming responsible for a major production on their own. Each of these specialists 
must work within their own design language that becomes aligned with the main plan for the entire dance 
production itself, thereby necessitating the precise nature of that plan. 
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History of Dance Notation 

The world of dance is characterized by competition and creativity. The notation system was a key factor in 
the evolution of the field. Notation systems can be textual, hieroglyphic, geometric, or a combination thereof. As 
dancers advance in the study of ballet, they learn a notation system that provides a public, portable record of 
movements in a dance they are learning. 

The history of dance notation begins with a pervasive oral tradition of ballet. In 1588 Arbeau published 
Orchesographie, which only notated movements of the arms and the body. The number of symbols was few because 
dances were limited to only a few movements and patterns in those days. The dance step symbol was written next to 
the note to which the movement corresponded on the accompanying musical score. This notation was meaningless 
to the unversed, but it was useful to dancers, and was in use after 200 years of its invention (Watts, 1998). 

The founding of the first dance academy, by Louis XIV, served as a stimulus to institute a notation system 
when the need to create and communicate production information increased. From this point on, ballet flourished 
and the variety and artistry of dances developed rapidly. This period saw an age of shorter costumes, softer shoes, 
expressive use of arms and body and a greater precision of the timing of movements. This, in turn, provided an 
increasingly sophisticated variety of dances to be recorded. Several choreographers attempted to use stick figures to 
notate their dances. These systems were the forerunners of modem notation systems, represented today by the 
Labanotation and Sutton Methods; two among dozens of notation systems that have been invented. 

Notation systems and design languages 

So far in this paper, we have been describing notation systems. We have also described how notation 
systems are essentially symbolic systems for representing designs that are conceived in a design language. 
Specialized design languages of the choreographer, the stage manager, and other production staff members must 
align with the dancers’ language and the notation system makes the choreographers’ design language expressions 
public so that coordination and synchronization can occur and as a means of advancing the complexity of production 
design. It is important to distinguish the notation system, which is public and concrete, from the design language, 
which is personal and abstract. 

What is the relationship between abstract design languages and their concrete terms and notation systems? 
“Written musical notation is not music design language. It is simply a good visible representation of the terms and 
relations that exist in the musical design language in the mind of the composer” (Gibbons, work in progress) and in 
this case the choreographer. “Notation systems give imperfect but visible expression to design structures; they grow 
with — and help to grow — the design language itself (Gibbons, work in progress).” 

[The] mutually supportive relationship between the abstract design language as it exists in the 
designer’s mind and the notational system used to express designs publicly is a very important 
one, and it is the reason for discriminating between a design language and its visible, public 
representation. The notation system becomes the first necessity for a design language to grow and 
mature beyond its often vague and limited rudiments. Until there is a system of notation (even a 
primitive and completely personal and idiosyncratic one), the precision of designs, and the 
expression of increasingly complex and nuanced designs is not possible, and designers become 
captured by the process of manufacture (Gibbons, work in progress). 

Design language - notation system interactions 

The evolution and elaboration of dance notation systems was the visible outcome of an increase in the sophistication 
and detail of the abstract design languages emerging in the choreographers’ minds. The notations were useful tools 
for the choreographer to communicate dance plans to dancers, production workers, and other choreographers. But 

the design languages and the notation systems were both capable of independent 
change and improvement. This is because of a mutually beneficial resonance that is set- 
up when symbol systems are used to externalize abstract designs. As designers improve 
and extend their personal design languages, this in turn calls for extensions and 
improvements to the notation system (See Figure 1). The notation system then is 
capable of expressing more interesting and complex designs and easily leads to 
innovation. This interaction between notation system and design language was a 



gives new terms to 




Design Notation 
Language System 




gives new terms to 
Figure 1 Cycle of Improvement 
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primary vehicle for the blossoming of Western music in all its variety following approximately 1000 A.D. 

Once a consistent notation system is established, if it is used with discipline, it can become; (1) a tool for 
remembering designs, (2) a structured problem solving work space in which designs can take form, and (3) a kind of 
laboratory tool for sharpening and multiplying abstract design language categories. It permits ever more complicated 
and interesting structures to be built and new dimensions of the design language, such as musical harmony, to be 
considered and explored in detail. Moreover, as these experiments yield new insights into the design language, they 
find expression through corresponding growth of the notation system. Thus, through a continuing cycle of 
refinement, both design language and notation system grow together in parallel, and more sophisticated 
technological ideas can result. 

“The qualities of a notation system, have major impact on the rate of growth of the related design 
language. Prove this to yourself by first multiplying two roman numerals and then multiplying the 
same quantities using Arabic numerals. It is not hard to see the great facilitation to designing 
numerical expressions and computations that comes from our current number system and its rules 
for place value representation. Without these both, algorithmic multiplication would be much 
more cumbersome. This example and others that could be given constitutes a rich field of 
exploration for those who would design notation system principles and design languages 
(Gibbons, work in progress).” 



Dance notation basics 

When analyzing a dance notation system, there are basic elements to look for. It is vital to remember the 
needs of the dancers, choreographer, stage designers, lighting crew, director, producer, and stage hands. It is 
important that all involved with the production are able to follow the dance score and adjust their activities in 
support of it. For instance, it is imperative that the stage designer, lighting crew, director, and producer know the 
interplay of the participants, so that they can design the stage, props, and organize the lighting so as to make the 
performance run flawlessly. A dance notation system is the common public language that all players possess with 
which they communicate and coordinate their individual plans. 

The basic elements of a good notation system would include symbols representing dancers’ positions on the 
floor, arm, leg, and head direction and movements, interactions with additional dancers, moods of the movement, 
measures, speed, and any applicable musical notation including time signatures, repeats, timing, and measures. 
Although both methods of dance notation included in our study contain these vital elements of a good notation 
system, we have observed important differences between the systems that may teach us something about the 
characteristics of a good notation system of any design field. For the remainder of this paper we will compare these 
two systems and draw out principles we feel transfer to the work of instructional designers. 

The Labanotation System 

Kinetographie was a notation system for recording human movement originally developed by Rudolf von 
Laban. It was modified in the U.S. by many, including Ann Hutchinson Guest, to become what is currently referred 
to as Labanotation. 



Left Rlgtt 



i= 5 upport column 
2=leg gesture column 
3=body column 
4=arm column 
5=head column 



Figure 2: Labanotation basic staff structure 
(after Griesbeck, 1996) 



Labanotation is written on a staff consisting of three vertical 
lines. Figure 2 shows this staff in solid lines. All movements 
performed on the right side of the body are noted on the 
right side of the staff, and all movements performed on the 
left side of the body are noted on the left side of the staff 
The dotted lines, not usually shown in dance notations, 
represent subdivisions of the body and symbols are placed 
in these columns to show movements and positions of the 
legs, body, arms, and head, respectively. The score is read 
from bottom to top (Griesbeck, 1996). 

This method of notation is thorough and quite accurate. 
Figure 3 is a composite of several detailed standards of this 
system. The rectangular diagram on the left of figure 4a 
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(reading from bottom to top), shows a notation sequence of three basic dance positions, using Labanotation. The 
positions being performed are shown on the right of figure 4b. Figure 5 shows how the length of symbols 
corresponds with musical notation timing. 

Once a dancer acquires the advanced skills necessary for Labanotation, he/she has a tool with which to 
record many forms of human movement including sports movements, like ice skating and martial arts. This method 
of notation becomes complicated and confusing with the addition of extra parallel staffs and symbols for additional 
dancers. Additional information that must be recorded for complete works are body position, timing, direction, 
position, arm and leg gestures, center of weight, transference of weight, phrases, and floor plans for all dancers in 
coordination. For instance. Figure 6 is a simple example of one musical measure (4 beats) of a ballet movement for 
one dancer. Once all the elements of the dance are in place, the score is difficult to read. 
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Figure 5: Labanotation versus musical 
notation (Griesbeck, 1996) 







Figure 6: Four measures of ballet 
in Labanotation (Griesbeckj 1996) 



The Sutton Dance Writing Method 

“Sutton Dance Writing is a new, international notation system similar to music notation. Just as music uses 
notes on a five-lined staff to record sound, Sutton Dance Writing records choreography on a five-lined staff (Sutton, 
2002).” This notation system was invented by Valerie Sutton and first published in 1973. It is a hieroglyphic system, 
which moves stick figures across a page like caricatures, creating an animation-like sense of movement (see Figure 
7). Each line on the staff corresponds to a specific height of the dancer with respect to the floor. The bottom line 
depicts the feet, second line the knees, third line the hips, fourth line the shoulders and the top line depicts head 
height. As shown in Figure 8, this notation system combines the abstract dance movements with the visual by 
placing animations on the staff. A different perspective is provided by symbols below the stick figure drawings, 
showing the overhead view as if looking down on top of the head (upper row of circles), and seeing the limbs 
projecting in various positions (lower row of circles) (Sutton, 2002). 

The Sutton Method is intuitive and easy to learn. A novice can look at a score and have a general idea of 
the movements. Figure 9 depicts the movement across the stage (represented by the square on the left of the staff), 
body movements (including the direction the dancer faces), feet placement and the tempo of the dance (represented 
by “1 & 2 &”) (Sutton, 1983). 

Dancers find The Sutton Method easier, and many academic dance programs are now teaching this method. 
This notation system gives as much information as the more difficult Labanotation, but many people are more 
comfortable working within the more graphic Sutton Method. 
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Figure 7: Basic Sutton Method staff 
(Sutton, 2002) 
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Figure 8: Basic ballet move using 
Sutton Method (Sutton, 2002) 
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Figure 9: Three measures of a ballet in Sutton Method 
(Sutton, 2002) 



Notation Summary 

The comparison of these two notation systems provides important insights that instructional technologists 
should find interesting. These insights relate to the dimensions on which notation systems vary and criteria for 
which a new notation system can be chosen or constructed. We believe these principles apply to notation systems in 
general and definitely to ones for instructional technology. We want to emphasize one of those dimensions that 
appears to have the greatest implications for the widespread adoption of notation systems; we call this the intuitive 
versus non-intuitive dimension. The most striking contrast between the Laban and Sutton notation systems is the 
clear resemblance between the Sutton notation system and the dancers’ mental model of a dance compared to the 
almost complete lack of resemblance for the Laban notation system. 

A notation system, like Sutton Notation system, can use diagrams to approximate what a person sees with 
their eyes, and therefore, to what a person might imagine while designing. The Labanotation system is the 
equivalent of a different language from our natural visual language, which doesn’t look anything like the final 
product. It doesn’t look anything like the experience that is created and envisioned by the designer. So the first 
lesson from the comparison of these two notation systems is that a notation system will, to some degree, match the 
experience that it is intended to represent, and the designer’s mental image of it, thus facilitating comprehension and 
eliminating needless mental translations. 

Examples of notation systems that look like what they are intended to represent are blueprints and drawings 
of the physical structures that are used by architects and those with whom they work. In chemistry, multiple notation 
systems have been invented to capture and represent chemical composition and structure. In organic chemistry there 
are four notation systems that serve different purposes; some of them very useful for designers of new compounds 



450 



and some of them almost entirely useless in design. There is, first of all, the verbal expression of the chemical name 
such as, sucrose. This notation has no inherent clues as to the composition and structure of the chemical itself. A 
second chemical notation system gives the atomic number of different elements such as, 22^11 ’ without 

showing the structural arrangement of the atoms. This requires less translation than the first example. A third 
chemical notation system is graphic and uses geometric drawings mixed with textual symbols to denote its benzene 
rings, chains, and other physical structures (see Figure 10) (Oregon State University, 2003). In this system, there are 
rules for drawing specific kinds of structures, and once again certain geometric information is gained and a certain 
amount of information lost, for example, the specific name of the chemical. Finally, there is a notation system which 
uses atom spheres fastened together in a 3-D configuration (see Figure 1 1) (Indigo Instruments, 2002). The kinds of 
information that are available in these models include realism relative to the physical proportions of the atoms and 
possible chemical reaction sites. The computer can model chemicals and some of their behavior using this kind of 
notation. Working with such visible models backed up by a computer model gives the chemist a working shape of 
the atoms as well as information related to specific charges and reactions to other chemicals. And so the atomic 
model representation invites chemists to use this in design and invites them to contemplate what might happen if a 
reaction were to occur at a certain point. This is a relevant part of protein chemistry today. 




Figure 10: Sucrose composition 
(Oregon State University, 2003) 




Figure 11; Sucrose model (Indigo Instruments, 2002) 



We conclude from this example that notation systems can be related to phenomenal models and perhaps treated as 
simplified versions of them, thereby, making them intuitive. At the same time, a notation system can be computable 
as well as resembling the imagined reality. 

Dimensions of a notation system 

While reviewing the notation systems for dance we began to notice patterns that allowed useful 
comparisons between systems. We also reviewed additional notation systems in other disciplines including those 
used in chemistry, architecture, and instructional technology and came up with some suggested general categories 
for comparing notation systems. These can be considered dimensions along which notation systems vary. 

1 . Intuitive versus non-intuitive. Intuitiveness resulting from visual similarity to mental images, as discussed 
above, is one of the most important aspects promoting ready usefulness. An example of an intuitive 
notation system would include the blueprints used by architects and those who work on creating buildings. 
When a novice looks at the floor plan, they have a general idea of the basic layout of the house that it 
represents. Even a novice will know that the dining room is adjacent to the kitchen and that there will be a 
walk-in pantry connected to the kitchen. The foundation portion of a blueprint, on the other hand, is 
intuitive only to those who are experienced in seeing foundations before houses are placed on them. The 
average homeowner has trouble visualizing this seldom seen, but essential element of the house design. 

2. Complex versus simple. A complex notation system, like Chinese writing, has a steep learning curve. With 
a notation system including 50,000 characters, only about 5,000 characters are in common use. A 
newcomer to Chinese would spend countless hours acquiring the knowledge of speaking and writing skills 
to be able to thrive in Chinese speaking communities. In contrast, a simple notation system could be found 
in highway signage found within the United States. The Exit sign is easy to recognize in any state. 

3. Computable versus non-computable. The symbols of a computable notation system can be used by 
computers or other computational means to extend, modify, and execute designed actions or structures. An 
appropriate combination of computer software and a midi interface can translate traditional musical 
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notation into musical performance. It is possible to play a piece of music on a keyboard connected to a 
computer and have the computer not only record the piece digitally, but also record the notation and make 
it available for editing and distribution. Written English used to be a non-computable notation system. 
However, recent advances in software make it possible to speak English into a voice dictation system and 
then take the typed output and feed it into a text reading system which reproduces (usually at lower fidelity) 
most of the original voice patterns. (This imperfect correspondence between abstraction and public 
language only emphasizes the value of considering notation apart from abstract design languages.) Most 
notation systems are not computable, but notation system designers should realize the potential power of 
computability and consider it as they invent notation systems. 

4. Precise versus imprecise. A precise notation system is exemplified by the organic chemistry notation given 
before. An imprecise notation system is exemplified by the recording of foreign names and sounds in 
English, particularly when the foreign language uses a different character set, e.g. Cyrillic, Hebrew, 
Chinese, or Japanese, etc. Since there are no rules for the capture of foreign language syllables in English, 
notations of them vary widely in literature and news reporting. 

5. Rapid recording versus slow recording. Notation systems are especially instrumental in capturing events 
while they happen or during design. The Sutton system, because it involves drawings, and since the 
drawings resemble the visual scene allows for rapid notation and a recorder can get much, if not all, of a 
dance recorded while it is being performed. Even if minor details of a dance can’t be captured rapidly this 
way, the properties of the Sutton system allow major pivotal points to be captured and details to be filled in 
later. This quality of this notation system compliments the rhythm of the design recording process, allowing 
major features to be recorded rapidly and minor details to be added later. This is a design metaphor 
resembling the “tweening” metaphor used in animation systems. Details tweened can include minor 
movements, nuances, and emotion and mood content (see #1 1 below). Several factors contribute to a slow- 
recording notation system: non-intuitive shapes and symbols, complex constructions, and large and 
intricate sets of notation symbols. Egyptian and Mayan hieroglyphic notation systems are examples of this. 

6. Resembles the sensory event versus does not resemble the sensory event. When an element of a notation 
system resembles sensory elements of the experience being designed, then fewer mental translations are 
required during both capture and execution of the design. A notation system that exemplifies this is the 
storyboarding process of making an animated film. Storyboards give enough of an overview so that 
newcomers to the project will be able to look at the “snapshots” and get a general idea of the storyline. The 
more a notation system resembles the actual event the less room the person using the notation has for 
personal interpretations. Labanotation is an example of a system that does not resemble the event it depicts. 
The dance notation in no way resembles what the choreographer sees in his/her mind. Although with study 
the notation system can be interpreted to produce the intended dance, a novice would not find the system to 
resemble the event and many mental translations are required. 

7. Support for improvisation versus no support for improvisation. Some notation systems anticipate some 
degree of improvisation, that is, they omit some portion of an otherwise detailed notation. In Handel’s 
Oratorio works, small notations above the musical staff indicate points for the soloist to “have at it” for as 
long as they choose, in improvising additional non-annotated notes. Another example of a notation that 
supports (or encourages improvisation) is jazz. Jazz notation includes the basics of the song; the improvised 
parts are not included. Measures are left for use by various soloists, but the solos are not written out and in 
many cases not rehearsed beforehand. A notation system that offers little or no support for improvisation 
includes mathematics. 

8. Interpretable versus non-interpretable. Interpretation is the act of varying a performance from the exact 
specifications of the noted design particularly by applying values not constrained by the designer in the 
notation. An example of a notation system that is interpretable is the Sutton Dance method of recording 
dance. The system offers caricatures that give an outline of the performance. The system allows for some 
interpretation by the dancer and allows the choreographer, dancer, director, and others to add their own 
subtle changes to the score in order to present their own intended meaning. An example of a notation 
system that is non-interpretable is a blueprint for a precision artifact of any kind. Manufacture of an 
airplane hydraulic actuator, the pistons that move the airplane’s control surfaces, are considered completely 
authoritative. Any interpretations added to the noted design by a machinist would be severely reprimanded. 

9. Representative of roles versus not representative of roles. A musical score, a play script, the choreographic 
plan for a ballet, and (we may speculate) and instructional design, frequently notate their designs for the 
production roles taken by multiple separate team members. Not only must such notations show individual 
parts of a performance/artifact manufacture, but they must also show the synchronization of those parts. 
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10. Context aware versus non-context aware. In some cases, design notation must indicate the correlation 
between multiple art forms or technology forms. For instance, a dance notation must show correlation with 
the orchestral score. Instructional technology designs also must sometimes show coordination with other 
ongoing design activities. A company that wishes to integrate training events and activities with normal 
workflow must show how workflow and training events correlate. 

11. Inclusion of emotional content & mood versus no inclusion of emotional content. One notation system that 
includes emotional content and mood is sign language. People express anger, sadness, joy, love, and other 
emotions through sign language, and the notation system used in training manuals for sign language often 
includes notations of emotional content and mood shown by facial expression or body motion. Musical 
scores also posses an entire subgroup of symbols describing the degree of passion a performer should 
exhibit while playing a particular passage. 

State-of-the-art of design languages and notation systems in instructional technology 

Instructional designers currently use a great variety of non-standard notation conventions to communicate 
their designs: flowcharts, storyboard forms, scripts, diagrams, sketches, and text descriptions. But there is little 
standardization among or between these conventions. This situation is analogous to that during the formative period 
of western musical notations. Crosby (1997) describes the evolution of musical notation from 900 A.D. to 1300 
A.D. He tells the story of the invention of numerous idiosyncratic systems that varied the number of staff lines, the 
shapes of musical notes, the representation of timing, and correspondence with some portions of the performance. 
Over time, the best features of these have been retained in the notation system in common use today. This does not 
mean that there are not still alternative notation systems, for there are many normally used for experimental musical 
ideas. 

We believe a similar notation convention can emerge for instructional designers, but it may be many years 
or even decades before this happens. One force that may hasten this process, making it shorter than that for music, is 
the computer, particularly the tool of semantically oriented mark-up languages like XML and its many descendants. 

Some volume-production design systems create categories of message or strategy elements that give 
structure to form-based approaches (Horn, 1974; Merrill, 1983). However these systems focus mainly on a particular 
layer of a design (Horn: content structures; Merrill: strategy structures) and do not approach the completeness of a 
blueprint system. 

Instructional technologists may look to other fields such as chip design, or computer programming designs 
where notation systems have moved to a higher level of sophistication (Miller, 2002). Notation systems in these 
areas have matured from sketch-edged to straight-edged. In these fields there has also grown up a shared 
terminology and standard context for the sharing of designs. Whereas, blueprints used to be printed on very large 
paper, 3-dimensional airplane and automobile designs are now displayed on computer screens (Petroski, 1998; 
Rifkin, 2002; Sabbagh, 1996). The complexity possible in designs in these areas has increased greatly, and greater 
variations in design are possible because these designs are expressed in languages and notation systems that are 
compatible. The speed of designs and the transfer of designs directly to manufacture have been greatly facilitated. 

Perhaps the most important outcome of a focus on design notation systems will be a focus also on the 
design languages that underlie them. Design languages are implicit in every design field. For instructional designers 
they relate to rules and principles for the generation of instructional experiences: “These rules. . .are the rules of an 
instructional grammar. Eventually we should develop generative grammars for instruction”(Stolurow, 1969). 

Through attention to design languages, and their related notations systems, designers can approach designs 
with more versatile and precise tools. Moreover, a medium of expression, comparisons, and evaluations of designs is 
opened. “A design language is like a natural language, both in its communicative function and in its structure as an 
evolving system of elements and relationships among those elements” (Winograd, 1996). 

If a language is what people use for communicating information and ideas to each other, then a design 
language is what designers use to communicate designs, plans, and intentions to each other. Further research into 
design structure, design languages, and design notation systems will better equip designers to create powerful, more 
precise designs. 



Summary 

Instructional technologists should be interested in notation systems because design involves more than one 
person. Notation systems capture designs to be reviewed, modified, maintained, and built by someone besides the 
designer. Given a common language for design, the process of communicating is less complicated and easier to 
understand by all those involved in the design. 
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This paper illustrates a number of dimensions that can help designers form their design languages and 
select the characteristics of their notation systems. It is certain that even with a limited vision of design languages 
and notation systems, great variation can emerge, testing many alternative approaches to publicizing designs. From 
this abundance of ideas we expect that notation standards for instructional designs can emerge just as did those for 
music and that we will discover that the complexities of instruction can be represented in terms of a modest set of 
powerful primitives. 

Within the field of instructional technology we must begin to ask questions about the usefulness of notation 
systems. As in the example of dance productions it is impossible that one person posses the needed design skills in 
all of the aforementioned areas. It is our assertion that if we adopt a common notation system or language, we will 
be better equipped to deal with publishing instructional designs into the common work space which will allow better 
communication with others of the instructional design team including programmers, graphic designers and other 
instructional designers. 
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An Integration of Situated Cognition and Interaction Theory in Designing 

Web-based Learning Environments 
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Abstract 

Building learning environments that create and maximize learning opportunities has been a major concern 
in designing web-based environments. The majority of learning theories guiding web-based learning are based on 
constructivist principles, which emphasize the role of an active learner. Situated cognition and interaction theory 
have been highlighted as contributing to a broader understanding of the role of an active learner. The purpose of this 
paper is to outline how assumptions of situated cognition and interaction theory can be utilized in a constructivist 
perspective to design web-based environments that promote learning. 

Introduction 

Web-based learning has been the form of distance education highlighted in recent years. In order to build 
effective and efficient web-based learning environments for students, researchers have applied various learning 
theories to practice. Early forms of web-based learning relied on the recreation of course content to a web format 
using HTML and Hypertext. Such systems focused on information transmittal. However, there is a move toward 
designing web-based learning environments based on principles of constructivist learning theories. 

The most promising constructivist theories are those that integrate principles of situated cognition and interaction 
theory to promote learning. The purpose of this article is to investigate the possible implications of situated 
cognition and interaction theory in designing web-based environments and to apply them to the actual development 
of a web-based learning environment. 



Assumptions of Constructivism 

Constructivist theory explains learning as an attempt by the learner to build his or her knowledge. Learning 
should occur in complex, realistic, and relevant environments that support multiple perspectives and the use of 
multiple modes of representation. Additionally, the learner should engage in social negotiation as an integral part of 
learning (Driscoll, 2000). In comparison, the learning process in situated cognition focuses on embedded learning in 
a realistic environment. Moreover, in interaction theory the learning process focuses on social interaction or 
exchange between two or more people or groups. Therefore, situated cognition and interaction theory can provide 
valuable suggestions for designing web-based learning environments from a constructivist perspective. 

Situated Cognition 

In their model of situated cognition. Brown et al. (1989) suggest that meaningful learning is only taking 
place when it is embedded in the social and physical context within which it will be used. Additional support for 
situated cognition comes from research in the area of transfer. Transfer is more likely to occur when the individual 
learner has an opportunity to develop knowledge and skills in an environment that is similar to the one in which he 
or she will eventually perform (Bassock, 1990). Moreover, Barwise & Perry (1983) argue indexical representation 
plays a critical role in developing an understanding of an environment, that is, certain items must be present in the 
environment in order to understand it. Lor example, they argue it is difficult for a third party to understand a two 
party conversation when pronouns, which lack immediate representation, are used. Suchman (1987) describes the 
potential problem of using diagrams in a manual to gain an understanding of how a particular object is put together. 
Lrom the perspective of situated cognition, it is not only a realistic environment that is of importance, but also the 
content tool use that occurs in such environments. Working in a realistic environment permits one to gain critical 
experience with the content tools. Equally important, the content tools found in the environment may influence how 
a learner thinks about and behaves in that environment. One might argue that these tools of the environment act as 
indexical representations cueing the learner’s potential actions or performances. More simply put, the tools may 
help structure the activity. 

Situated cognition assumes that learning occurs through increased participation in communities of practice. 
Wenger (1998) explained that knowing is a matter of participating in the pursuit of active engagement with the 
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world; knowledge is developed through the lived practice of the people in a society. These communities of practice 
can be structured in learning communities or cognitive apprenticeships. The implication for instruction that is 
suggested by cognitive apprenticeships, learning communities, and anchored instruction is the importance of a 
situated context for solving complex and realistic problems. This assumption of situated cognition coincides with the 
constructivist’s requirement for realistic and relevant environments. From the viewpoint of instructional designers, it 
is necessary to design learning environments in which students can participate actively and deal with real problem 
situations for exposure to multiple representations of the material. This active participation may occur between 
individuals or between individuals and the tools of their environment. A second learning theory, interaction theory, 
buttresses the implications gained from the situated cognition perspective. 

Interaction Theory 

Interaction theory supports the constructivist perspective of the learning process. Learners construct 
knowledge through social interaction (Vygotsky, 1978). Social interaction provides mediated interpretations of 
experiences from each student. While individuals perceive and understand the world in different ways, some overlap 
in understanding may be created. Through this, individuals can learn other points of view and build a complex 
understanding of the world. Symbolic exchanges like language enable the individual student to achieve a common 
understanding, create meaning, and internalize as thought external processes (Vygotsky, 1978). These assumptions 
of interaction theory coincide with the support for multiple perspectives within constructivist theory. Multiple 
perspectives can provide the grounds for various forms of scaffolding that can maximize learning in the Zone of 
Proximal development for a given student in a given content area (Vygotsky, 1978). The interaction has to occur 
among students or between student and expert. However, Vygotsky also argued that tools could also serve as 
mediators of thought. By manipulating tools in one’s environment and using tools as a form of scaffolding for 
learning, tools possess the potential for helping internalize various forms of thought. Therefore, instructional 
designers need to consider the use the synchronous chatting, asynchronous e-mail, or bulletin boards as additional 
tools. From the viewpoint of an instructional designer, it is crucial to design learning environments that promote the 
sharing of multiple perspectives. What, then, are the implications of these theories to the web-based learning 
environment? 



Designing web-based learning environments based on the implications 

The web is a computer network crossing a geographic area, constructing social and cultural communities 
through a network. Students can construct their own meaning through this space. To achieve the goals of situated 
cognition, instructional designers need to consider how to provide students with authentic contexts and generate 
authentic activity in such contexts. This is potentially the greatest challenge to those designing learning 
environments in a web-based setting. Certain content areas may be more readily adaptable to this form of delivery 
system, and it is important for an instructional designer to first determine what he or she desires as learning 
outcomes in his or her students (Rothwell & Kazanas, 2000). Once these learning outcomes have been established it 
is necessary to list the strategies and tactics that could be used to facilitate the learning outcomes. Based on an 
analysis of learning outcomes and strategies/tactics, one should then determine which are feasible for a web-based 
delivery system. While there is no specific algorithm for arriving at this decision, a general approach will take into 
account the potential strengths of web-based instruction. These include flexibility, content expandability, learner 
autonomy, and collaborative learning (Jung, 2001). Moreover, web-based instruction offers the ability to structure 
activities that foster problem solving and higher level thinking skills. Students must apply knowledge and skills to 
solve problems in the contexts of real practice. Unlike other types of web-based learning such as tutorials, this 
approach is not a fact-oriented design. It encourages learners to solve authentic problems in which knowledge will 
be developed and utilized. Moreover, tools related to successful performance in a real world setting should be 
incorporated into the activity. 

If problems support multiple solutions, students may need to work from multiple perspectives when 
problem solving. Therefore, a second method to generate authentic activity is to provide students the opportunity for 
collaboration. Collaboration supports a knowledge construction process among learners. Learners can explore, 
discuss, and integrate different views through the expression of multiple perspectives, reflection, peer review, and 
feedback. Collaboration could be structured as student and student, student and group, and group and group formats. 
While the interaction in situated cognition focuses on participation as a member in a community of practice, a 
second approach, interaction theory, focuses on cognitive development through scaffolding and coaching. The role 
of instructional designers would be to provide enough complexity of a real-life context and relevant authentic 
activities. To do this, a variety of resources that do not simplify the problem are needed. For example, students may 



456 




need access to relevant multiple websites or expert’s websites for gathering information, and such tasks should be 
group-based to encourage collaboration. Additional web-based tools, independent of content tools, are also needed 
to support their collaboration such as schedules, archives, self-monitoring functions, whiteboards, chatting rooms, 
etc. To achieve goals of interaction theory, the web can be used to enable social interactions between student-student 
and student-expert through chatting, e-mail, and bulletin board. These are web tools to support the exchange of 
individual thought. The tools also provide students the opportunity to assist other students with scaffolding through 
synchronous and asynchronous communication. Moreover, web-based tools may be developed to pair less 
knowledgeable learners with more knowledgeable others in order to provide the necessary scaffolding to encourage 
knowledge construction. 



Table 1 Situated Cognition/Interaction Theory Guidelines for Application 

Situated Cognition/Interaction Theory 
Guidelines for Application 

Decision Points 

□ E - In what environmental context will the learner have to ultimately perform? 

□ K - What knowledge or skills will be required for optimal performance? 

□ A - What activities can be provided to the learner so that he or she can develop the knowledge and 
skills associated with the intended environmental context? 



Analysis of Environmental Context: Potential for Transfer 

Describe the relevant participants, activities, goals, and tools associated 

with the environmental context in which the learner is expected to perform. 



□ 

□ 

□ 

□ 


Participants: what are the potential roles of the individual? 

Tasks: what are the tasks in which student may be involved? 

Goals: what are the desired performance outcomes of the context? 
Tools: what tools or artifacts may be associated with the environment? 


Analysis of Knowledge and Skills: KSA’s for Transfer 


Describe the relevant knowledge forms and skills associated with performance of 


the tasks in the environmental context. 


□ 


Remembering content knowledge (declarative) 


□ 


Planning 


□ 


Deriving solutions 


□ 


Making decisions 


□ 


Communicating 


□ 


Developing attitudes 



Determine if the individual needs to be able to state how he or she performs the skill (either while 
within or external to the environmental context) or if successful performance is sufficient. This is 
a potential component of assessment. 



Analysis of Instructional Activities: Methods for Transfer 

Describe the instructional activities to be used and how they meet the criteria for 
situated cognition or interaction theory. 

□ Activity allows for collaboration 

o Between students 
o Between instructor and students 

□ Instructor models thinking processes 

o Instructor provides verbal descriptions of thinking processes 
o Instructor relates thinking to contextual circumstances 
□ Activity context closely approximates actual environmental context 
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o 


Students provide activity/environmental context 


o 


Instructor provides activity context 

■ Simple: Instructor provides clear interpretation. 

■ Complex: Instructor requires students to develop an interpretation of 
the context, ensuring students possess relevant prior knowledge. 


□ Activity 


incorporates contextually relevant tools 


o 


Tools are shared as a group 


o 


Tools are held individually 


□ Activity 


allows students to practice skills 


o 


Students practice skills in a collaborative setting 


o 


Activity requires students to verbalize practice 



Despite the fact that the web affords possible ways to accomplish the assumptions of these two theories, 
one must continue to examine how real world contexts may be represented according to situated cognition and to 
consider types of interaction needed to manifest the assumptions of instructional theory as a instructional designer. 
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Situated Cognition/Interaction Theory 
Guidelines for Application 



Decision Points 



□ E - In what environmental context will the learner have to ultimately perform? 

□ K - What knowledge or skills will be required for optimal performance? 

□ A - What activities can be provided to the learner so that he or she can develop the knowledge and skills 
associated with the intended environmental context? 

□ 

Analysis of Environmental Context: Potential for Transfer 

Describe the relevant participants, activities, goals, and tools associated 
with the environmental context in which the learner is expected to perform. 

□ Participants: what are the potential roles of the individual? 

□ Tasks: what are the tasks in which student may be involved? 

□ Goals: what are the desired performance outcomes of the context? 

□ Tools: what tools or artifacts may be associated with the environment? 



Analysis of Knowledge and Skills: KSA’s for Transfer 

Describe the relevant knowledge forms and skills associated with performance of 
the tasks in the environmental context. 

□ Remembering content knowledge (declarative) 

□ Planning 

□ Deriving solutions 

□ Making decisions 

□ Communicating 

□ Developing attitudes 

Determine if the individual needs to be able to state how he or she performs the skill (either while within or 
external to the environmental context) or if successful performance is sufficient. This is a potential 
component of assessment. 



Analysis of Instructional Activities: Methods for Transfer 

Describe the instructional activities to be used and how they meet the criteria for 
situated cognition or interaction theory. 

□ Activity allows for collaboration 

o Between students 
o Between instructor and students 

□ Instructor models thinking processes 

o Instructor provides verbal descriptions of thinking processes 
o Instructor relates thinking to contextual circumstances 

□ Activity context closely approximates actual environmental context 

o Students provide activity/environmental context 
o Instructor provides activity context 

■ Simple: Instructor provides clear interpretation. 

■ Complex: Instructor requires students to develop an interpretation of the context, 
ensuring students possess relevant prior knowledge. 

□ Activity incorporates contextually relevant tools 

o Tools are shared as a group 
o Tools are held individually 

□ Activity allows students to practice skills 

o Students practice skills in a collaborative setting 
o Activity requires students to verbalize practice 
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Using Bloom’s Taxonomy to Determine the Academic Level of Learning 

Objectives 



Chris Winter 
Brian Lindquist 
University of Phoenix 

Abstract 

This author proposes an extension to Bloom ’s taxonomy in order to more accurately dijferentiate between 
undergraduate- and graduate-level learning objectives. The extension proposed is a categorization of conditions 
present during the performance of skills specified in learning objectives. Bloom’s taxonomy and the proposed 
condition categories will form an x-y axis; the dissection of this axis will enable a more objective determination of 
the academic level of learning objectives. 

Using Bloom’s Taxonomy to Determine the Academic Level of Learning Objectives 

Bloom’ s taxonomy has been widely recognized by the education community as a standard in the 
determination of academic rigor since its development by Dr. Benjamin Bloom and his colleagues during the period 
1948 - 1952. The taxonomy consists of six steps, or levels, of student learning ranging from knowledge through 
evaluation (Bloom, 1956). While Bloom’s taxonomy provides an indication of academic rigor, it has yet to be 
shown how the taxonomy can be used as a tool to differentiate between undergraduate and graduate learning 
objectives. In this paper, an extension to Bloom’s taxonomy is proposed that will enable more accurate 
differentiation of the desired outcomes of undergraduate and graduate courses. 

The Need 

Undergraduate and graduate courses differ primarily in the performances expected of students, rather than 
the body of knowledge covered. In fact, graduate-level course materials are often indistinguishable from their 
undergraduate counterparts. Normally, the assignment and assessment items are the tools used to distinguish a 
course as graduate level. Due to the direct link between learning objectives and assessment. Bloom’s taxonomy can 
be used as a tool for differentiating graduate from undergraduate courses; a certain number or percentage of 
objectives from different Bloom’s levels can be required. For example, at the University of Phoenix it has been 
determined that 80% of the objectives in graduate courses need to describe skills from levels four, five, and six of 
Bloom’s. Some questions raised by this utilization of Bloom’s taxonomy are “who decided on this percentage’’ and 
“how did they come to this decision?” There really was no objectively defensible method for assigning these 
percentages; it was just an arbitrary decision. What is needed is a second dimension to Bloom’s Taxonomy that will 
take the guesswork out of undergraduate/graduate distinction. 

Background 

Questions regarding the adequacy of Bloom’s Taxonomy as a determination of academic level began 
occurring with the examination of different learning objectives from undergraduate and graduate courses containing 
the same Bloom’s performance verb. The conditions stated in these learning objectives have as much, if not more, 
impact on the overall academic level of the objectives as the performance verbs themselves. For example, consider 
the following objective: “The student will calculate the value of a bond.” The performance verb for this objective is 
“calculate;” it is found at level three of Bloom’s taxonomy - application. At the application level, students must be 
able to apply knowledge or skills to new situations in order to solve problems, answer questions, or perform other 
tasks (Bloom, 1956). Now, compare the objective stated above with the following application-lex/ &\ objective: “The 
student will calculate the value of a bond during a period of financial instability.” This objective clearly calls for the 
application of a different set of knowledge and skills. However, the Bloom’s level of these two objectives is the 
same; that is, they both describe application-level skills. How, then, can Bloom’s Taxonomy be useful for 
determining academic level? 

Conditions Categories and Measurement Levels 

Learning objectives are traditionally divided into the following three sections: performance, conditions, and 
criteria (Dick, Carey, & Carey 2001). Performance is indicated by the verb contained in the objective (e.g.. 
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demonstrate, explain, compare and contrast). Criteria refers to the accuracy, speed, quality, etc. with which students 
are required to complete the performance (e.g., with no mistakes, with 50% accuracy, within 5 minutes). Conditions 
are any materials required to perform the stated task. Conditions in a learning objective may be physical equipment 
or resources such as a computer, a diagram, a model, etc., or they may be environmental (e.g., during a period of 
financial instability). As demonstrated by the bond example above, the conditions stated in a learning objective can 
be very important in determining the academic level of the objective; however, there currently is no method for 
classifying these conditions in order to utilize them, along with the performance, to determine the academic level of 
objectives. Next, we will explore the feasibility and ramifications of classifying conditions into general categories 
and subdividing each category into measurement levels. Table 1 presents the three general categories proposed 
along with their measurement levels. 





Resources 


Information 


Environment 


HjH Inclusive resources 


Complete information 


Static environment 


Missing resources 


Incomplete information 


Changing environment 



Tab 



e 1. 



Condition Categories and Measurement Levels 



Resources 

The resources category represents tools, systems, processes, and equipment required in order for students to 
perform the skills specified in learning objectives. Resources is the most tangible and easily identified and adjusted 
of the three categories. Moreover, the lack of resources is arguably easier to accommodate for when performing a 
task than is lack of information and instability of the environment; therefore, it is the lowest-level category on the 
proposed scale. It is not necessary, here, to specify which items will be included within the general category of 
resources; this will be left as general as possible in order to accommodate a wide variety of items and to increase 
flexibility for the designer. However, some subdivision is needed in order to identify the measurement levels within 
this category. 

The first measurement level within the resources category is inclusive resources. What is meant, here, by 
inclusive resources is that all the resources required to complete the task will be given to the student. For example, 
when calculating the value of a bond, the student will be given the necessary formulas and/or methods for 
performing this task. Because of the “spoon-fed” nature of this measurement level, it is the lowest level within the 
general category of resources. 

The second measurement level of the resources category is missing resources. Missing resources means 
the student must complete a task without having all of the required tools, systems, etc. For example, when 
calculating the value of a bond, not every needed formula is provided to the student. This incompleteness of 
resources requires the student to improvise and make use of critical thinking and decision-making; skills much in 
demand in the workplace. Due to the real-world nature of this measurement level, it is the highest level within the 
resources category. 



Information 

The information category represents knowledge, training, beliefs, and attitudes that students have at their 
disposal when performing the skills specified in learning objectives. Information, unlike resources, is intangible and 
difficult to identify and adjust. Furthermore, while performing a task is difficult with missing resources, the 
difficulty is greater while performing with limited information; therefore, information is at a higher level than 
resources on the proposed scale. Generally, information is considered to be any data processed and given meaning 
and/or context; however, as with the resources category, there is no need to specify which items are to be included in 
the information category (beyond this general definition). Again, as with resources, two measurement levels in the 
information category are proposed. 

The first measurement level within the information category is complete information. What is meant, here, 
by complete information is that all the information required to complete the task will be given to the student. For 
example, when calculating the vale of a bond, the student will be given specific information regarding the bond and 
the procedure for calculating its value. Because the student is merely required to follow instructions in this 
measurement level, it is the lowest level within the general category of information. 
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The second measurement level of the information category is incomplete information. Incomplete 
information means the student must complete a task without being given all the required information. For example, 
when calculating the vale of a bond, the student is not given all of the operational and procedural information for 
doing so. This incompleteness of information requires the student to recall and synthesize prior knowledge and 
beliefs into the performance of the task. Due to the synthesis of prior knowledge involved in this measurement 
level, it is the highest level within the information category. 

Environment 

The environment category represents external influences acting upon students when they perform the skills 
specified in learning objectives. Environment is external to the student, and, for the most part, independent of 
influence by the student. Furthermore, performing a task in a changing environment is more difficult than with 
missing resources or incomplete information. Therefore, environment will be considered as the highest-level 
category on the proposed scale. As with resources and information, there will be two measurement levels within the 
environment category. 

This first measurement level within the environment category is static environment. What is meant by 
static environment is that the influences acting upon the student are constant during the performance. When 
calculating the value of a bond, the Federal Interest Rate remaining constant from one period to another would be an 
example of static environment. Static environment does not require students to adapt for a changing environment; 
therefore static environment is the lowest level within the environment category. 

The second measurement level within the environment category is changing environment. A changing 
environment, here, is one in which the influences acting upon the student are dynamic; that is, they are changing 
quickly enough to require adaptation on the part of the student in order to compensate for them. In other words, the 
outcomes generated by student actions (i.e., the performance objectives) will be significantly altered if no adaptation 
occurs. Since, arguably, a dynamic environment is the most challenging of our measurement levels, it will be 
considered the highest of these levels. 

The Undergraduate/Graduate Axis 

Application 

The condition categories and measurement levels developed in the previous section of this paper along with 
the levels of Bloom’s taxonomy form an x-y axis (Figure 1) that can be used to more objectively differentiate 
between undergraduate- and graduate-level learning objectives. In order to use this axis, the designer will simply 
plot the point on this axis representing the intersection of the Bloom’s and condition levels of the objective. Where 
this point lies on the axis will indicate whether the objective is undergraduate- or graduate-level. 

Use of this axis takes much of the guesswork out of determining whether a course is undergraduate- or 
graduate-level. Rather than requiring an arbitrarily-assigned percentage of objectives from various levels of 
Bloom’s taxonomy, the designer can require that all of the objectives in a course fall within either the undergraduate 
or graduate section of the axis, depending upon the intended audience. 

Drawbacks 

Of course, the use of this axis will not preclude any judgment on the part of the designer. Some 
subjectivity will still be inherent, at times, in determining the categories and measurement levels under which the 
conditions stated in learning objectives fall. 

Also, judgments will have to be made during situations in which the performance/condition position plots 
on the line dividing graduate and undergraduate. For example, if the objective contains a performance at the 
comprehension level of Bloom’s and the condition stated indicates incomplete information, then the 
performance/condition intersection lies directly on the line delineating undergraduate from graduate. In situations of 
this nature, the designer will have the following two options: make a judgment call, or revise the learning objective. 

A third potential difficulty in using this axis arises when learning objectives do not contain explicit 
conditions statements. While some consider conditions to be a required component of learning objectives, there is 
disagreement among designers as to the utility or value added to a course by this requirement. However, if no 
conditions are stated in a learning objective, it can be assumed that the implicit conditions are inclusive resources, 
complete information, and a static environment. In other words, if no conditions are explicitly stated, then the 
performance alone will determine the academic level (i.e.. Bloom’s levels one, two, and three will be undergraduate 
while levels four, five, and six will denote graduate-level skills). 
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Summary 

This paper has introduced a tool for using Bloom’s taxonomy to more objectively differentiate between 
undergraduate- and graduate-level learning objectives. The tool consists of an x-y axis made up of the levels of 
Bloom’s taxonomy and the conditions stated in learning objectives. While designers using this tool will still need to 
make judgment calls in some situations, they will at least have some assistance in making these decisions. 
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Relationships between Promoting Self-Regulatory Skills And Facilitating 
Student Interactions in Online Learning Environments 

Yong-Chil Yang 
Andong National University 

Abstract 

The use of self-regulatory skills (SRS) and student interactions have become as major research issues for 
successful and effective online learning. The purpose of this study was to investigate the effects of student 
interactions on promoting student SRS in online learning environments. Research questions are whether design 
strategies to facilitate student interactions have impacts on promoting the use of student SRS, or vice versa. It was 
expected that student interactions in the environments could contribute to the acquisition or use of their SRS. 32 
college students were taught in online learning environment for 16 hours over 8 weeks. With 2 hours per week, one 
hour was assigned to students ’ content learning, and the other one for group discussions, having four members each 
group. SRS questionnaires were administered before and after online learning. Results from this study showed that 
embedded SRS strategies and group discussions in online learning environments have a possibility of contributing 
to the acquisition and use of their SRS. In the light of such findings, instructional design strategies to promote 
student SRS and to facilitate student interactions in online learning environments were discussed. 

Introduction 

It is well known that learners are responsible for their own learning. It means that "learning is not something that 
happens to students; it is something that happens by students"(Zimmerman, 1989, p. 21). To have this happen, it 
seems to be crucial that students have some extent of their self-regulatory skills (SRS) before learning. However, it 
is likely that most of public school students or many of college students would not come into learning settings with 
appropriate skills of regulating their own learning. 

Recently, universities are moving more and more towards flexible modes of delivering courses. While high 
school has traditionally been a face-to-face experience, post secondary education is limited to its contact time and is 
being increasingly channeled through information and communications technology or Internet resources (Roblyer, & 
Wiencke, 2003). This fading of close social interactions in college significantly diminishes the regularly 
mechanisms that ensure students’ natural development (McMahon, & Oliver, 2001; Chou, 2003). The lacks of close 
contact between instructors and learners or among learners prevent learners from developing their self-regulated 
learning. Lidner and Harris (1993) provided some evidence that self-regulated learning are an important part of 
academic success for college students. Peverly et al. (2003) showed that college students were not as good as 
expected at self-regulation. It means that many of college students need self-regulation training. But they had few 
opportunities to become self-regulated in their secondary school years, and as a result, they have few even if any 
SRS strategies (Orange, 1999). In particular, it has known that student SRS are required to be successful in online 
learning (Cennamo, & Ross, 2000; Cho, 2003). Accordingly, it seems that design strategies to promote the 
acquisition or use of student SRS are main research issues in online learning environments. Also, it is expected that 
design strategies to facilitate student interactions contribute to promoting acquisition or use of student (MacGregor, 
& Atkinson, 2002-03; Oliver, Omari, & Herrington, 1998). 

Interaction seems to be an important factor in online learning environments as well as in face-to-face learning 
settings (Dykes, 2001; Hillman et al., 1994; MacGregor, & Atkinson, 2002-03). Especially, it is shown that learner 
interactions can become a good strategy for building learning environments that entails learners expressing their 
knowledge, getting feedback and constructing knowledge. More generally, learner interactions in online may be 
useful in light of pedagogical goals (Craig et al., 2000; Vrasidas, & Mclsaac, 1999). Cazden(cited in Dytes, 2001) 
summarized the cognitive benefits of learner interactions: (1) learners are forced to confront each other; (2) learners 
can enact complementary roles, provide mutual guidance and support, and can service as scaffolding to help each 
other accomplish learning tasks that might otherwise be too difficult; (3) learners can find a direct relationship with 
a real audience from which they can get meaningful feedback; and (4) learners experiment and construct new 
understanding and ideas in peer discourse setting. 

It is important that learners know when they need to ask help and be willing to ask for or to accept assistance 
from someone in learning settings. Orange(1999) suggests that such self-regulation as asking help, giving feedback. 
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self-monitoring, making elaboration and organization can be taught and that peer models or cooperation can be 
effective means for teaching self-regulation. Therefore it is expected that student interactions in online learning 
environments contribute to the acquisition or use of their SRS. More recently, researchers reported that promoting 
students’ SRS is possible by their own efforts such as student interactions or cooperation (Craig et ah, 2000; Hartley, 
2001; Orange, 1999). If then, it appears significant to investigate the effects of student interactions on promoting the 
acquisition or use of his/her self-regulatory skills in online learning environments. 

The purpose of this study was to investigate the relationship between promoting student interactions and 
facilitating SRS in online learning environments. In order to explore any relationship between these two, design 
strategies to embed some components of SRS and student interactions through discussions need to be applied to 
authentic online learning environments. It is assumed that the degree of promoting the acquisition or use of learner 
SRS might determine the amount and depth of his/her learning in online learning environments, especially when 
designing web-based learning environments. 



Methods 

Participants 

32 undergraduate students from one university in Korea participated in the study. Participants were enrolled 
in a two-credit required course that is Introduction to Educational Technology. There were 8 males and 24 females, 
all junior students with ages ranging from 21 to 24, respectively. They were divided into 8 groups, and each group 
has 4 members. They were randomly assigned to each group because they are comparatively familiar with each 
other to induce their active participation at group discussions 

Materials 

The course for this study was developed into online learning materials. It was taught under online learning 
environments. The amount of learning materials is for eight weeks with 2 contact hours per week. Participants 
logged in on the site and learned the course once a week for 8 weeks. Some of strategy components to promote SRS 
during learning from online were embedded into the contents of the course. 

The learning activities in each lesson were designed to encourage learners to study the materials in online and to 
enable them to deeply explore a given topic through group discussions every class meeting. The order of the 
presentation was lesson objectives, outlines, content, summary, preview of the next lesson, and discussion about 
given topics. There were also five quizzes included in each lesson. Discussion between group members was made 
for 50 minutes, and then each group submitted its result to the discussion task room in an online environment. 

Embedded SRS Strategy Components 

Strategies to Promote The Use of SRS 

Strategies for promoting learner SRS were embedded into the learning materials. SRS strategy components 
employed in this study were performance control strategies (self-instruction & self-monitoring), self-efficacy 
strategies (peer feedbacks & attribution feedbacks), and cognitive strategies (elaboration & organization). These 
SRS strategies were selected because learners have hardly acquired or comparatively used them less than any other 
strategies during their learning. The embedded SRS strategies in learning materials are the following: 

Performance Control Strategies (Zimmerman, 1998): 

(a) Self-instruction: Telling for themselves in mind to memorize major concepts or key principles. Or 
students learned by reciting important concepts for themselves. 

(b) Self-monitoring: Checking out for themselves the results of their own learning activities 
Self-efficacy Strategies (Zhang et ah, 2001 ) 

(a) Peer feedbacks: Encouraging or praise each other about peer ideas or comments from group discussion, 
or active participation during discussion. 

(b) Attribution feedbacks: Providing effort feedbacks for prior success supports students’ perceptions of 
their progress, sustains motivation, and increases self-efficacy for learning (Schunk, 2001). They were 
encouraged to give attribution feedbacks to each other during discussion. 

Cognitive Strategies (Park, 2000; Cho, 2003): 

(a) Elaboration: Paraphrasing with learners’ own words, and link new information to what they already 
know when learning key words or concepts in the text. 

(b) Organization: Summarizing the contrast, and developing a table or a figure mentally. They were asked 
to summarize their learning materials on the provided board. 
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Figure 1 Text screen with embedded SRS strategies 



Strategies to Facilitate Student Interactions 

(a) It focused on group communications through synchronous discussion. Each group selected one of the 
given topics through discussion. 

(b) Student interactions were performed through discussion to explain and solve the topic selected by the 
group members for around 50 minutes per week. 

(c) Discussion was synchronously made through the MSN messengers in an individual computer screen. 
Group members logged in after learning the tasks in online. 

Group members were encouraged to use attribution feedbacks to each other as often as possible during 
discussion, for instance, using expressions like “You work very hard!” “You’re very knowledgeable on that issue!” 
when they have good idea during discussion. Also, the use of praise or encouragement words like “good job” or 
“very well” was encouraged among group members. And feedback examples were exhibited during discussion 
sessions. 
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Figure 2. An online discussion screen 

Measuring Instruments 

SRS questionnaires adapted by Yang (2000) and Park (2000) from the original versions of Zimmerman and 
Martinez-Pons (1986), and Pintrich and De Groot (1990) were used to measure the acquisition of student SRS 
before and after online learning. The 84 item self-report measure addressed participants’ SRS. Among the 84 items, 
items of performance control strategies were 14, self-efficacy strategies 11, and cognitive strategies 13. The attitude 
survey was composed of 15 items, and was administered right after learning. The frequencies and features of 
interactions between learners and the results of discussion on the given topics were collected. 

Research Design 

Research design used in this study was pretest-posttest control group design. While this design is one of the 
experimental designs used extensively when applying new methods or programs in education to instruction, it falls 
far short of handling the other sources of external validity (Tuckman, 1999). In the present study, the design was 
adopted because of exploring the effects of online learning programs on the acquisition or use of SRS through 
student interactions. The independent variables in the study were the embedded SRS strategies and student 
interactions during online learning and the dependent variables were scores yielded from student SRS measure, and 
responses to the attitude survey. 

Procedures 

The study was conducted as a part of students’ regular classes for pre-service teachers training. The 
participants completed the SRS measure as a pretest right before the beginning of the study. They learned the course 
contents in an online environment for 8 weeks with two contact hours per week. Learning activities were divided 
into two parts. The first part was the online learning activities and the second was discussion activities among group 
members about given topics through the MSN messenger. Immediately after an eight-week study schedule, they 
took the SRS measure as a posttest, and the attitude survey. 

Data Analysis 

Data yielded from SRS measure were analyzed into mean scores on the pretest and posttest between the 
experimental and control group. The t-test was used, and the level of a set at .05. Results of the attitude survey were 
analyzed by contrast with SRS results. And interaction processes by each discussion group were examined. 
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Results and Discussions 

The effects of Embedded SRS Strategy Components 

The effects of embedded SRS strategy components and student interactions through discussion on the 
acquisition or use of SRS were examined in online learning environments. To determine whether there was a 
difference between the pretest and posttest scores yielded from SRS measure on the experimental group, a t-test was 
used. As shown in Table 1, the mean difference between pretest scores (M = 3.44, SD = .46) and posttest scores (M 
= 3.60, SD = .44) in the experimental group was not statistically significant, t (31) = 1.669, p > .05. This means that 
the use of embedded SRS strategy components and student interactions in online learning environment could not be 
useful in facilitating the acquisition or use of student SRS in this study. 



Table 1 Means and SD for Scores of embedded-SRS strategy components 



Test 


n 


M 


SD 


t 


df 


P 


Pretest 


32 


3.44 


46 








Posttest 


32 


3.60 


44 


1.669 


31 


.105 



The Effects of The Whole SRL Measure 

Although not statistically significant, the embedded SRS strategies and student interactions in online 
learning did appear to have substantially positive impact on the acquisition or use of student SRS as compared to the 
result of the control group. As shown in Table 2, the mean difference between posttest scores of the experimental 
and control group (M = 3.60 and 3.57) was far smaller than the mean between pretest and posttest means (M = 3.44 
and 3.60) on the experimental group. This means that the use of embedded SRS strategies and student interactions in 
online learning could contribute to promoting the acquisition or use of student SRS (Cennamo & Ross, 2000; 
Hartley, 2001; Peverly et al., 2003), or to facilitating student interactions through the use of his/her SRS (Dykes, 
2001; Orange, 1999). 



Table 2 Means and SD for the total SRS strategy scores by each group 



Group 


n 


M 


SD 


t 


df 


P 


Experimental 

Control 


32 

35 


3.57 

3.53 


.44 

.37 


.419 


65 


.677 



The Results by Each of the Embedded SRS Strategy Components 

It appears that the results were mainly yielded from group discussions in online learning because the mean 
difference between pretest scores and posttest scores for performance control strategies was larger than other 
embedded SRS strategy components (see Table 3). This means that interactions among students might be an 
important factor influencing the acquisition or use of SRS strategies. When they discussed about a topic with group 
members through online MSN messengers, 78% of participants (32) said that student interactions were useful to 
promoting self-monitoring during learning. Although both the use of self-efficacy strategies and cognitive strategies 
were improved during online learning and student interactions, none of them was statistically significant in the 
degree of improvement. 

Table 3 Means and SD by each of embedded SRS strategy components in online learning 







Pretest 




Posttest 






n 


M 


SD 


M 


SD 


Performance Control Strategies 
(Self-instruction & self- 
monitoring) 


32 


3.36 


.67 


3.59 


.59 


Self-Efficacy Strategies 

(Peer feedbacks: immediate and 

attribution) 


32 


3.34 


.49 


3.47 


.48 
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Cognitive Strategies 
(Elaboration & organization) 



32 3.47 .55 3.67 .54 



Relationships of SRS Strategies and Student Interactions in Online Learning 

On the other hand, the mean scores of the whole SRS strategies on the pretest and posttest of the 
experimental group are likely to show that the possibility of facilitating student SRS strategies through group 
interactions in online learning is much higher. That is, the mean scores of the whole SRS strategies on the pretest 
and posttest were further lower than mean differences on the acquisition or use of embedded SRS strategies on the 
experimental and control group (see Table 1 and Table 4). This suggests that student interactions through discussions 
among group members through the MSN messengers (student interactions) could influence promoting the use of 
student SRS strategies (McMahon, & & Oliver, 2001; Zhang et ah, 2001), or that the acquisition or use of student 
SRS strategies might have an impact on facilitating student interactions in online learning. 



Table 4 Means and SD for the whole SRS on the experimental group 



Test 


n 


M 


SD 


t 


df 


P 


Pretest 


32 


3.51 


.40 


.630 


31 


.533 


Posttest 


32 


3.57 


.41 



However, it is recognized that this study had several limitations, which could not gain more positive impact 
on the acquisition or use of student SRS strategies in this study. For instance, the study had comparatively short 
periods in online learning (McMahon & Oliver, 2001), an inappropriateness of embedded SRS strategies design 
(Cennamo & Ross, 2000), and the weak validity of SRS questionnaire to be measured in online learning (Orange, 
1999). However, the results of the present study suggest that the acquisition or use of student SRS strategies in 
online learning environments can be facilitated through embedded strategies in lesson materials and student 
interactions using synchronous discussions. Further research on promoting the use of student SRS or facilitating 
student interactions is required to examine the effectiveness of each components of self-regulatory skills in online 
learning environments. 



Conclusions 

This study was conducted to investigate the effects of embedded self-regulatory skills strategies and student 
interactions through group discussions in online learning environments. The results from this study showed that the 
embedded SRS of performance control strategies, self-efficacy strategies, and cognitive strategies did not influence 
the acquisition or use of student SRS in online learning. Those strategies suggest that when compared to the whole 
SRS strategies, however, they could promote using student SRS. This means that promoting the use of SRS might 
be facilitated by student interactions through online group discussions. This study seems to have some implications 
for providing a possibility of exploring relationships between promoting SRS and facilitating student interactions 
which are necessary in designing and developing an effective online learning environment. 

References 

Berge, Z. L. (1998). Guiding principles in web-based instructional design.. Educational Media 
International, 35(2), 72-76. 

Berge, Z. L. (1999). Interaction in post-secondary web-based learning. Educational Technology, 39(1), 

5-11. 

Cennamo, K., & Ross, J. (2000). Strategies to support self-directed learning in a web-based course. Paper 
presented at the Annual Meeting of the American Educational Research Association. New Orleans, LA. April 
24-28, 2000. 

Chou, C. (2003). Interactivity and interactive functions in web-based learning systems: A technical 
framework for designers. British Journal of Educational Technology, 34(3), 265-279. 

Cho, M-H. (2003). The effects of design strategies for promoting self-regulated learning on students’ self- 
regulated learning skills and achievements in WBL environments. Unpublished thesis. Andong National University, 
Andong, Korea 



469 




Craig, D., ul-Hag, S., Khan, s., Zimring, C., Rick, J., & Guzdial, M. (2000). Using an unstructured 
collaboration tool to support peer interaction in large college classes. In B. Fishman & S. 0’Connor-Divelbiss(Eds.), 
Fourth International Conference of the Learning Science{pp. 178-184). Mahwah, NJ: Erlbaum. 

Dykes, M. (2001). Assessment and evaluation peer interaction using computer-mediated communication in 
post-secondary academic education, http://www.usask.ca/education/coursework/802papers/dvkes/dvkes.htm 

Hartley, K. (2001). Learning strategies and hypermedia instruction. Journal of Educational Multimedia and 
Hypermedia, 10(3), 285-305. 

Hillman, D., Willis, D., & Gunawardena, C. (1994). Learner-interface interaction in distance education: An 
extension of contemporary models and strategies for practitioners. The American Journal of Distance Education, 
8(2), 29-42. 

Lidner, R., & Harris, B. (1993). Teaching self-regulated learning strategies. Paper presented at the 
convention of the AECT, New Orleans, LA. 

MacGregor, S., & Atkinson, T. (2002-03). Lacilitating learner interaction in the two-way video classroom. 
Journal of educational Technology Systems, 31(1), 45-62. 

McMahon, M., & & Oliver, R. (2001). Promoting self-regulated learning in an on-line environment. In ED- 
Media 2001 World Conference on Educational Multimedia, Hypermedia & Telecommunications. Proceedings 
(O**"), Tampere, Einland, June 25-30, 2001 

Oliver, R., Omari, A., & Herrington, J. (1998). Exploring student interactions in collaborative world wide 
web computer-based learning environments. Journal of Educational Multimedia and Hypermedia, 7(2), 263-287. 

Orange, C. (1999). Using peer modeling to teach self-regulation. The Journal of Experimental Education, 
68(1), 21-39. 

Park, H-K. (2000). The effects of different ways of employing self-regulated learning strategies in CBI. 
Unpublished dissertation. Florida State University, Tallahassee, FL 

Peverly, S., Brobst, K., Graham, M., & Shaw, R. (2003). College adults are not good at self-regulation: A 
study on the relationship of self-regulation, note-taking, and test taking. Journal of Educational Psychology, 95(2), 
335-346 

Pintrich, P., & De Groot, E. (1990). Motivational and self-regulated learning components of classroom 
academic performance. Journal of Educational Psychology, 82, 23-40. 

Roblyer, M., & Wiencke, W. (2003). Design and use of a rabric to assess and encourage interactive 
qualities in distance courses. The American Journal of Distance Education, 17(2), 77-98. 

Shin, 1998 

Schunk, D. (2001). Social cognitive theory and self-regulated learning. In B. J. Zimmerman, & D. H. 
Schunk(Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (2nd ed., pp.191-226). 
Mahwah, NJ: LEA 

Tuckman, B. (1999). Conducting educational research(5'^ ed.). Eorth Worth, TX: Harcourt Brace 

Vrasidas, C., & Mclsaac, M. (1999). Factors influencing interaction in an online course. The 
American Journal of Distance Education, 13(3), 22-36. 

Yang, M. H. (2000). The study on development and validation of a self-regulated learning model. 
Unpublished doctoral dissertation, Seoul National University, Seoul 

Zhang, J., Li, F., Duan, C., & Wu, G. (2001). Research on self-efficacy of distance learning and its 
influence to learners’ attainments. Unpublished research paper. Tsinghua University, Beijing, China. 

Zimmerman, B. J. (1989). Social cognitive view of self-regulated academic learning. Journal of 
Educational Psychology, 81, 329-339. 

Zimmerman, B. J. (1998). Developing self-fulfilling cycles of academic regulation: An analysis of 
exemplary instructional models. In D. H. Schunk, & B. J. Zimmerman(Eds.), Self-regulated learning from teaching 
to self-reflective practice(pp. 1-19). New York: Guilford Press 

Zimmerman, B., & Martinez -Pons, M. (1986). Development of a structured interview for assessing student 
use of self-regulated learning strategies. American Educational Research Journal, 23, 614-628. 



470 



The Application of Instructional Design at the University 



Alan L. Young 
Brigham Young University 

Abstract 

Instructional technology can be applied in any setting where there is a need to improve performance and 
understanding. Because of historical and cultural traditions, universities do not typically apply instructional 
technology. This transition would require that faculty be trained in the various facets of instruction, or else a team- 
approach to the design and development of instruction must be applied. Recommendations for the application of 
instructional design at the university include any combination of the following: a clear definition of the functions 
and roles of the university and its personnel, new looks at faculty development efforts, a redefinition of the college 
classroom, and a change in faculty roles. These types of changes will allow universities to produce high quality 
learning experiences. 



Introduction 

In a recent meeting of university faculty, an administrator stood and explained that the university setting we 
are used to no longer exists. A fundamental change is that students (and their parents) are requiring more 
accountability of the university they choose to attend (and to which they give thousands of dollars each year). The 
climate is changing such that students are becoming more critical education consumers, requiring that the university 
they choose provide them with a certain level of knowledge and skill in exchange for their tuition money. 

This same university was recently reclassified and restructured from a junior college into a bachelor’s 
degree granting university. In the process of this transition, the board of directors for the new university stressed that 
“effective teaching and advising will be the primary responsibilities of its faculty, who are committed to academic 
excellence” (2000). 

Faculty at institutions of higher education are required to wear multiple hats which require that they be a 
professional subject matter expert, instructional designer, instructional developer and instructor. Principles of 
instructional design and the scholarship of learning and teaching often become secondary to research in the subject 
matter, and “teaching the way I was taught” becomes the standard. 

Improved teaching and learning at the university can occur by identifying specific roles and functions in the 
instructional design process and applying researched principles of sound instruction to each of these functions. This 
can occur in a number of ways including compensation restructuring, focused faculty development efforts, 
maximizing delivery media, and redefining faculty functions. 

Background and Context 

Some brief definitions will provide context for understanding the application of instructional design at the 
university: 

• Instructional Science — “Instructional science is concerned with the discovery of the natural principles 
involved in instructional strategies . . . which nature will reveal as a result of careful scientific inquiry” 
(Merrill, Drake, Lacy & Pratt, 1996). 

• Instruction Design — “Instructional design is a technology which incorporates known and verified learning 
strategies into instructional experiences which make the acquisition of knowledge and skill more efficient, 
effective, and appealing” (Merrill, Drake, Lacy & Pratt, 1996). 

• Instructional Technology — Using the definitions of instructional science and instructional design as a 
basis, we can define instructional technology as the application of the principles discovered through the 
science of instruction, or, instructional design in action. The Commission on Instructional Technology 
defines instructional technology as “a systematic way of designing, carrying out, and evaluating the total 
process of human learning and communications, and employing a combination of human and nonhuman 
resources to bring about more effective instruction” (Commission on Instructional Technology, 1970). 
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In theory, instructional design would be found in practice in any place where there is a need for people to 
change or to gain knowledge or skills. Since they are defined by science, these principles of instruction should be 
universally applicable in any setting where instruction occurs. In the corporate environment and in automated 
instructional environments, instruction is commonly designed according to established principles of effective 
instruction. In general, instructional design is most commonly applied in areas where instruction involves a team 
approach. For the team to be effective, even for the team to be able to operate, there must be a design document in 
place. 

Due to a number of factors, including the history and tradition of universities and the relatively new and 
just emerging nature of the field of instructional science and instructional technology, instructional design is not 
typically applied in the university setting. 

The primary functions of today’s universities are threefold: research, teaching, and public service. The 
purpose of research is to identify true principles or laws. Research provides a foundation or basis for teaching. A 
university is a natural setting for teaching because it is typically the home of research. Researchers that double as 
teachers work to educate and train learners regarding scientifically-proven principles. These students will go on to 
apply those principles in the workplace. This cycle continues over and over. As true principles come to light through 
research, the truth is taught to others and then applied in the world (see Figure 1). This is how we function and 
progress as a society. 




Figure 1 : University Cycle 

This research-teaching-application cycle functions in all areas, except within itself. Teaching and learning is 
also a field of study, put into practice most often at the university level. Whereas the mountains or valleys are the 
most common setting of application for research and teaching in the field of Geology, the university is the most 
common setting of application for research and teaching in the field of Teaching and Learning. However, the 
research-teaching-application cycle is rarely applied in this setting. Universities are required to fill the essential role 
of teaching in this progression cycle, yet in doing so they tend to ignore the research on how to teach and learn most 
effectively. Traditionally, the truth-seeking universities ignore the principles of truth discovered through research in 
filling their fundamental roles of teaching students and helping them apply true principles in the world. While 
excelling in the area of research, universities do not sufficiently teach others and help them into a position to apply 
true principles. Because of the lack of attention to teaching and learning, the university itself corrupts the very cycle 
that justifies its existence. 

An effective learning experience is a work of creation that must be planned first, and then carried out. In 
my experience as a university student, teacher, and faculty consultant. I’ve found that the emphasis is commonly on 
what is being taught and not on how it was being taught. The design of the instruction is carried out haphazardly or 
on the spot. Good or not, educational experiences will occur as learners interact with each other, teachers, and 
specific subject matter. The effectiveness of those experiences will be greater if they are planned for and designed in 
advance. 



Instructional Technology at the University 

If the teaching and learning process is still a major component in the cycle that defines the university, then 
more attention must be paid to applying the research done on teaching and learning in the university setting. 
Whether we define the individual components of this education and training as lessons, units, modules, classes, or 
something else, there is a significant amount of instruction (intentional and non-intentional) that takes place as one 
of the foundational purposes of the university. The better this instruction is designed the more effective it will be. 

Faculty members at most universities are given the challenge of filling various roles in the instruction 
process. They must be the subject matter expert, the instructional designer, the instructor, the instructional 
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developer, and the instructional evaluator. Most faculty are hired because of their abilities as a subject matter expert 
(researcher) and most have little experience in filling the other important roles in the instruction process. 

If we look at examples from a cross section of university teaching we can see how this is the case. 
University faculty that are interested in true teaching have always had to wear at least two hats. While wearing one 
hat the faculty member is an expert in their given discipline. While wearing the other hat the faculty member is an 
expert in the field of teaching and learning, or instructional design. Given that instructional design is a field of study 
all its own, teachers need to have a “double-major.” 

The goal of the instructional designer is to make scientifically-based decisions that make each moment of 
the learner’s interaction with the subject matter deliberate so that the learning experience is effective and efficient. 
All educational experiences, simple or complex, are composed of different functions. These functions are defined 
below by the names of the people who would carry out that function. 

The Subject Matter Expert — ^has a clear understanding of the content the learner is seeking to 

comprehend. This person is commonly the researcher or practitioner. 

The Instructor — seeks to help the learner in his quest for understanding of the content. 

The Instructional Designer — ^plans what interactions and experiences will occur between the learner, the 

content, and the instructor to provide for maximum learning. 

The Developer — constructs any physical pieces of the instructional experience required by the design. 

The Evaluator — points out areas of the instruction that are incomplete or breakdown in the process of a 

learner coming to an understanding of the content. 

In most university settings, one individual carries out all of these functions. In most cases the faculty 
member is a subject matter expert in his or her discipline, but not specifically trained as an instructional designer, an 
instructor, or in any of the other roles that are involved in a successful learning experience. If we compare the 
instructional team to a football team, the faculty member in the traditional sense is required to play multiple 
positions - quarterback, running back, linebacker, and defensive end - all at the same time. 

There are two approaches to making instruction more effective in the university setting. First is a team 
approach to instruction where an expert fills each of the roles listed above. This team approach is often used in 
distance education environments, which require more advanced technologies and more precise advance planning of 
instruction than is typical in a traditional classroom. This distance education approach can benefit educators in any 
setting. In arenas outside education, a team of inventors, engineers, and builders combine their unique skills to 
produce a quality product — i.e. the Pentium processor or the Golden Gate bridge. Within education, a team of 
instructional designers, subject matter experts, instructors, and developers can also combine their unique skills to 
provide a quality learning experience for university students. This is an approach to learning that is not common in 
the university setting. “It seems that in many tertiary institutions, the ratio of instructional designers (IDs) to 
teaching staff is often around 1 ; 100-200. This means that many staff have to grope in the dark on their own even if 
they need support.” (Tan, 2002) 

The second approach to making instruction more effective requires the college instructor to be multi- 
talented. The educational roles and functions listed above are still met, but by only one individual. In addition to the 
formal training already received (i.e. Physics or Sociology), the instructor must gain the skills to fill each of the 
instructional roles. In any educational experience, each of these functions is carried out, deliberately or not. 

Effective instruction occurs when each of the functions is carried out with deliberation, care, and skill. 

Instructional Design happens at a number of levels at the university. The global level is university- or 
college-wide, often called program or systems design. The design happens at the local level for a specific class, or 
unit, or module. For it to be effective, some element of design should exist at the macro as well as the micro level, 
which would include semester design, daily class design, and others. 

Instructional design is most frequently applied and researched in areas of “automated” instruction, or 
technology-enhanced instruction, like simulations or other computer-aided instruction, where the dynamic element 
of a live teacher is not present. Instructional design is least often intentionally applied when the dynamic teacher has 
control over all elements of the instruction. This view (and current practice) has the teacher filling all of the roles in 
the design and development of instruction, not by design but by default. Because the teacher plays such a 
foundational role in the whole instructional process at the university, it may help to analyze the role of the teacher. 
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Historically, either by necessity or because of ignorance to the science of instruction, teachers have been 
required to fill all instructional roles. Education has become centered around the teacher. This tradition has been 
carried on. The opposite view is that no teacher is necessary. There is much instruction that takes place with little or 
no human interaction or interference. History has shown that hybrid models are the best. This is instruction that 
combines the best elements of face-to-face instruction with the best elements of automated instruction. This type of 
setting allows the teacher to do what they can uniquely do as a human, and letting the other parts of the instruction 
happen automatically. 

In the early days of instructional science, B. F. Skinner said: “There would be no point in urging these 
objections if improvement were impossible. But the advances which have recently been made in our control of the 
learning process suggest a thorough revision of classroom practices and, fortunately, they tell us how the revision 
can be brought about. . . . 

“The modern classroom does not, however, offer much evidence that research in the field of learning has 
been respected or used. . . In the light of our increasing knowledge of the learning process we should, instead, insist 
upon dealing with those realities and forcing a substantial change in them. 

“Education is perhaps the most important branch of scientific technology. It deeply affects the lives of all 
of us. We can no longer allow the exigencies of a practical situation to suppress the tremendous improvements 
which are within reach. The practical situation must be changed. 

“There is a simple job to be done. The task can be stated in concrete terms. The necessary techniques are 
known. The equipment needed can easily be provided. Nothing stands in the way but cultural inertia. But what is 
more characteristic of America than an unwillingness to accept the traditional as inevitable? We are on the threshold 
of an exciting and revolutionary period, in which the scientific study of man will be put to work in man’s best 
interests. 

“Education must play its part. It must accept the fact that a sweeping revision of educational practices is 
possible and inevitable. When it has done this, we may look forward with confidence to a school system which is 
aware of the nature of its tasks, secure in its methods, and generously supported by the informed and effective 
citizens whom education itself will create” (Skinner, 1954). 

Into Practice: From Here to There 

This statement of B.F. Skinner was made in 1954, yet the same fundamental conditions exist in the formal 
education setting today. It is possible to move from the current system and tradition to a better system, and the 
transition is already beginning to occur. This transition can only happen through the application of instructional 
technology at the university. 

Define Functions and Roles 

In making the transition, a beginning step is to outline the functions and roles or outcomes of the university. 
A university may determine that one of their necessary functions is to teach students. The individuals responsible for 
designing and implementing the instruction would be paid according to the success or failure of that goal. If students 
meet learning objectives the staff gets paid more. If students fail to meet the objectives, the staff gets paid less. This 
may provide motivation to the teaching staff to pursue proven teaching techniques, or principles of instructional 
design. 

Another recommendation for making the transition to better teaching at the university is to clearly identify 
and define the roles of the faculty member. Faculty members commonly focus on their role as subject matter 
experts. By defining their additional roles as instructional designers, instructors, developers, and evaluators, faculty 
members gain a clearer idea of their potential as teachers. Their job description becomes more complete. This also 
provides focus for the second recommendation for making the transition, faculty development efforts. 

Faculty Development 

Faculty development efforts help faculty fill all of the educational roles. Faculty can receive training and 
consulting in the areas of instructional design, development, and evaluation, including proven teaching techniques 
and strategies. Because one of the primary purposes of the university is teaching, the language of instructional 
design must become part of the language of the university faculty member. In addition to the scholarship of Biology, 
Chemistry, and Fiterature, faculty members must become versed in the scholarship of Teaching and Teaming. As 
they do this, the traditional teaching methods, which are being proven as less-effective, will be replaced by proven 
methods of successful instraction. Most universities employ educational technologists that can provide this 
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consulting service. This training will provide an opportunity for faculty members to become scholars of teaching and 
learning. 

Related to this training and consultation is the need to provide resources for faculty to develop their own 
toolboxes of instructional design principles and proven teaching techniques. Most campuses have faculty 
development centers that can be a source for these resources. However, these efforts can only be effective if they 
address the heart of the learning issues and provide opportunities for immediate application. In this scenario, faculty 
members would assume the various roles in the instructional design and development process (subject matter expert, 
instructional designer, instructor, developer, etc.), and receive training in these various roles. Faculty tenure and 
benefits would be tied in to their ability to perform in these various roles. 

An extension of this idea involves instructional evaluation. Many universities conduct student evaluations 
on their courses, but few of these evaluations are used or even intended to improve teaching. Faculty could use 
course evaluations as a basis for improving teaching, which would lead them to ask the fundamental questions about 
what makes good teaching. Faculty would be asking those questions that are answered in the instructional 
technology research. The key would be to align the faculty that are asking the questions with the people or resources 
that have the answers. 

Redefine Classroom 

A third recommendation for transitioning from the current state of university teaching to the application of 
Instructional Design calls for a new and different look at the college “classroom.” What is the advantage gained by 
being together in a room with students? Are there classroom activities that could be carried out more effectively and 
efficiently online, to provide more time to the unique elements of human interaction? Can faculty utilize other 
teaching tools to provide focus to their teaching? A valuable element of the university experience is being together 
in the same place with other people pursuing similar interests. Something unique occurs as individuals are in the 
same physical space. Individuals can sense things in other individuals allowing them to make comments and 
judgments influenced by the group dynamic. A common problem with having haphazardly designed instructional 
experiences is that the mediums get misused. A teacher takes valuable group time to do clerical tasks - i.e., 
announcements, explaining assignments, or going through rote memory exercises. The real value of the faculty 
member is wasted. Instructional designers can identify that which is unique to the classroom, and what the faculty 
member uniquely brings to the learning experience and capitalize on it. “Of course the teacher has a more important 
function than to say right or wrong. The changes proposed would free her for the effective exercise of that function. 
Marking a set of papers in arithmetic - ‘Yes, nine and six are fifteen, nine and seven are not eighteen’ - is beneath 
the dignity of any intelligent individual. There is more important work to be done - in which the teacher’s relations 
to the pupil cannot be duplicated by a mechanical device. Instrumental help would merely improve these 
relations. . . . If the advances which have recently been made in our control of behavior can give the child a genuine 
competence in reading, writing, spelling, and arithmetic, then the teacher may begin to function, not in lieu of a 
cheap machine, but through intellectual, cultural, and emotional contacts of that distinctive sort which testify to her 
status as a human being” (Skinner, 1954). 

Changing Faculty Roles 

A fourth and final recommendation is to change the role of the faculty member. Rather than requiring the 
faculty member to fill all the jobs required for effective instruction, this model would make the faculty member 
solely responsible for what they do best, being the subject matter expert. Other trained professionals would be 
brought in to do instructional design, instructional development. A professional instructor or facilitator would be 
employed, along with a professional evaluator. Faculty would play their unique role - while other roles would be 
played out in other ways. 



Conclusion 

The culture of higher education is changing. Students have more choices as to how they will receive their 
education, and are requiring more accountability from those providing the education. Those in the university setting 
can provide the best possible education by practicing principles of instructional technology. 

References 

Commission on Instructional Technology. (1970). To improve learning. A report to the President and the 
Congress of the United States. Washington, DC; Commission on Instructional Technology. 



475 




Ricks College to Become Brigham Young University-Idaho. (2000, June 21). Ricks College Press 
Relations. http://www.byui.edu/News/NewsReleases/BYU-Idaho.htm (visited 2003, Oct. 3). 

Merrill, M.D., Drake L., Lacy, M.J., Pratt, J.A. (1996). Reclaiming Instructional Design. Educational 
Technology 36(5), 5-7. 

Skinner B.F. (1954). The Science of Learning and the Art of Teaching. Harvard educational review. 
24(l):86-97. 

Tan, J.K. (2002, Oct. 29). Advisory Instructional Design systems. ITFORUM@LISTSERV.UGA.EDU 
(visited 2002, Oct. 29). 



476 




Implementing Portfolios As Comprehensive Assessments 

Richard Yzenbaard 
Western New MexicoUniversity 

Abstract 

The purpose of this study was to analyze how portfolios can be implemented as comprehensive 
assessments. Portfolios provide today’s educators with a viable means of measuring student knowledge within the 
framework of the life experiences and the values of the learner. Despite the potential that portfolios offer, the 
intellectual and resource demands of developing a portfolio assessment strategy are immense. Successfully 
implementing portfolios as comprehensive assessments is complex and involves a myriad of issues. Three specific 
questions guided this paper: 

1. What factors influence assessment strategies that are multifaceted, collaborative and reflect a 

comprehensive view of educational progress? 

2. What factors influence the implementation of a comprehensive portfolio assessment plan? 

3. How can the portfolio process be effectively managed on an individual and school-wide level? 

Managing and planning the portfolio process is an ongoing process. A long-term commitment to a 

comprehensive reform plan is needed to ensure success. As part of this long-term commitment the schools 
specifically need to address the following areas: time and scheduling, roles and responsibility and technology 
support. When considering portfolio implementation, one must realize that by embracing alternative assessments, 
educators are beginning rather than ending a complex process. The goal of employing appropriate assessments 
represents a series of decisions that should be continually questioned and revisited. 

Implementing Electronic Portfolios As Comprehensive Assessments 

The purpose of this research was to understand the factors associated with implementing portfolios as 
comprehensive assessments. 

The questions guiding this paper are as follows: 

1 . What factors influence assessment strategies that are multifaceted, collaborative and reflect a 

comprehensive view of educational progress? 

2. What factors influence the implementation of a comprehensive portfolio assessment plan? 

3. How can the portfolio process be effectively managed on an individual and school- wide level? 

Portfolios as an assessment tool, present educators with an instrument for assessing higher order, complex 

skills; skills not easily measured on multiple-choice tests. Student involvement in the creation of their portfolio 
fosters a sense of ownership for their work, setting of their goals and self-evaluation of their progress. Portfolio use 
as part of classroom activities has been reported by 57% of the teachers in a study conducted by Henke, Chen and 
Goldman (1999). English and language arts (40% of the teachers in the study) lend themselves more readily to the 
use of portfolios for assessing student work than do math, science, social studies and other subject areas (25% of the 
teachers in the study). 

Not only does the subject area presently determine the use of portfolios; the grade level of students does as 
well. The use of portfolios to assess at least one content area occurs in 75% of the reported primary classrooms and 
60% of the reported intermediate classrooms. In high school, 41% of the high school teachers reported using 
portfolios in at least one subject area. As the grade level of students increases, the use of portfolios declines. The 
use of portfolios to assess student work overall, and specifically in English, language arts and mathematics occurs 
less among intermediate teachers than among primary teachers (Henke et ak, 1999). About 75% of all primary grade 
teachers and 60 percent of intermediate grade teachers used portfolios to assess skills in at least one content area. In 
contrast, 41 percent of high school teachers reported using portfolios in at least one subject area. Even within the 
elementary grades, teachers’ use of portfolios declined with the grade level of their students: intermediate teachers 
were less likely than primary teachers to report using portfolios to assess student work overall and specifically in 
English/language arts and mathematics (Henke et ak, 1999). 
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Conceptual Framework 

Baker, Freeman and Calyton (1990) argue that not having a clear mechanism for evaluating students’ 
complex thinking and problem solving skills impedes our instructional programs and also communicates to teachers, 
parents and students that these untested skills lack importance. In order to develop comprehensible mechanisms we 
need to develop clear and consistent working definitions of the terms employed in the assessment process. 

The term “authentic” in this study refers to assessments for which there are no pre-specified answer. As 

Nystrand (1997) states: 

Authentic questions are questions for which the asker has not pre-specified an answer . . . 

Dialogically, authentic teacher questions signal to students the teacher’s interest in what 
they think and know and not just whether they can report what someone else thinks or has 
said. (p. 120) 

In this paper, assessments looking for one right, pre-specified answer are referred to as traditional assessments. 
Nystrand (1997) states that: 

By contrast, a test question allows students no control over the 
flow of the discussion ... a test question allows only one possible 
right answer, and hence monologic, (p. 120) 

With these definitions as our guide we can explore the background of portfolios as comprehensive assessments. 

Historical Perspective 

The following table provides a historical perspective of how assessment practices have been utilized by 
various societal and political groups. This timeline begins 2000 B.C. and concludes 4000 years later. 

Table 1 Assessment Timeline 



Year Significant event 

2000 B.C. Chinese civil service exams were established in a move toward meritocracy and away from unfair 

preference in choosing civil servants. 

400 B.C. Socrates in Athens, Greece developed conversational methods to assess the positions described 

and defended rhetorically by students. 

1700s Early U.S. testing involved oral examinations conducted by faculty who use professional 

judgment to determine the quality of student performances. 

1830s Horace Mann first used written tests in Massachusetts and Connecticut. Mann’s 1846 Boston 

Survey were the first printed tests for large-scale assessment of student achievement in geography, 
grammar, history, philosophy, astronomy, writing and arithmetic. The Boston School Committee 
was shocked by perceived low student performance. Because these tests results were not used, the 
use of the survey was discontinued in 1847. 

1895 Joseph Rice conducted testing in a number of large school systems. The results were: (1) 

negligible differences in spelling scores regardless of amount of instruction time, and (2) large 
differences in math scores among schools. The recommendation was to develop and utilize 
standardized examinations. 

1900s Alfred Binet developed tests of general intelligence (g), to identify developmentally disabled 

children in Paris and to provide them with remedial education. Edward Thorndike devised and 
promoted objective tests to improve on classroom assessments. The U.S. Army’s alpha and beta 
tests, for literate and illiterate World War 1 recruits, were used to match personnel to military 
assignments. By the 1920s, standardized, norm-referenced tests began to be widely implemented. 

1933-1941 The Progressive Education Association emphasized the ideas of John Dewey and others regarding 
individualized, experiential education. Ralph Tyler’s Eight-Year Study in 300 colleges and 
universities of students from 30 high schools associated with progressive education found that the 
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students outperformed matched peers (n = 1,475) who had been exposed to rigid college- 
preparatory curricula. 



1957 



1965 



1969 



1970s 



1980s 



1990s 



1994 



1994 



The successful launching of the first satellite, Sputnik 1, by the USSR generated intense desire 
in the United States to outperform Cold War competitors. Political momentum swelled for 
developing better instructional programs, especially in science and technology, and for comparing 
educational results internationally. 

The Elementary and Secondary Education Act (ESEA) of 1965 was passed, “new math” and 
other curricular and pedagogical innovations initiated, and educational evaluation established to 
study the effectiveness of the new programs. 

The National Assessment of Educational Progress (NAEP), promoted by Tyler and mandated by 
Congress, began to chart trends in student achievement using matrix-sampled tests. Directed by 
the National Center for Educational Statistics (NCES), constructed-response items were included 
in 1988. In 1990-1991, trial state assessments were first conducted. Eederal law named the 
National Assessment Governing Board (NAGB), a group of representatives of public and 
educational constituencies, as the NAEP policy-setting body. NAEP increasingly influenced state 
assessment programs. By 1998, the Educational Testing Service (ETS) and National Computer 
Systems (NCS) were heavily involved in NAEP. 

Standardized, norm-referenced (NRT), minimum competency tests were adopted in many states 
and districts across the United States. Subsequently, concerns arose about teaching to the 
minimum, and drop out rates increased. 

Use of criterion-referenced tests (CRTs) grew in reaction to problems with NRTs, as did emphasis 
on educational excellence and higher-order thinking skills. Opposition to standardized testing 
grew, but so did the public popularity of the tests. Publication of A Nation at Risk (National 
Commission on Excellence in Education, Washington, D.C: Governmental Printing Office) in 
1983 ushered in a new round of educational panic and reform fervor, which remained unabated for 
15 years. Reform focused not on curricular improvement but on accountability, with aggregated 
scores on standardized achievement tests as the prime indicator of educational quality. 

Testing dominated educational reform despite substantial research documenting problems with 
standardized tests, including biases, distortion of curriculum, and diversion of resources from 
instruction to testing. As alternatives to standardized tests, performance assessments were 
developed locally and by states. In 1989 the National Council of Teachers of Mathematics 
(NCTM) developed well-received standards, leading to the development of standards in other 
subjects by other organizations. States moved into standards -based assessment. Conservative 
opposition, centered on cost and privacy issues, effected retreats from performance assessment in 
some states. 

Improving America’s Schools Act (IAS A) mandated national biennial assessment of individual 
students and charged NAGB with determining which subjects to test, performance standards, 
instruments, and reporting results. A new national system of assessments, to include performance 
items, was to be voluntary. 

GOALS 2000: Educate America Act established national educational goals requiring that, by year 
2000: (1) all children will have access to preschools that prepare them for school; (2) 90% of 
students will complete high school; (3) all students will demonstrate competence in prescribed 
subjects in grades 4, 8, and 12; (4) all teachers will have access to professional development; (5) 
U.S. students will be first in the world in achievement in math and science; (6) all adults will be 
literate and able to compete in a global economy; (7) all schools will be free of drugs, alcohol, 
and violence; and (8) every school will promote parental participation in children’s development 
and education. 
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2002 



No Child Left Behind Act of 2001. George W. Bush signed into law the No Child Left Behind 
Act on Jan. 8, 2002. This act declares that all states must implement statewide accountability 
systems, which will: 

• Set academic standards in each content area for what students should know and be able to do, 

• gather specific, objective data through tests aligned with those standards, 

• use test data to identify strengths and weaknesses in the system, 

• report school academic achievement to parents and communities, 

• empower parents to take action based on school information 

• direct changes in schools that need help. 

How Traditional Assessments Relate to Authentic Assessments 

Assessments that allow multiple right answers and authenticate a student’s response have been regularly 
employed in reading and writing environments. Authentic assessments have the potential to allow for individual 
student differences and represent a more collaborative approach to assessment. Authentic assessment’s collaborative 
approach address improvement, effort and achievement while traditional assessments focus primarily on 
achievement (Tierney, Carter, and Desai, 1991). 

In portraying a negative perspective on traditional assessments Kohn (2000) contends that schools whose 
focus is to raise standardized test scores tend to abandon learner-centered environments to not only teacher-centered, 
but more destructively to a legislature-centered environment. 

In advocating for the use of authentic assessment for reading and writing Tierney et ak, (1991) make the 
following comparisons: 

Table 2 Comparisons between Authentic and Traditional Assessments 



Authentic Assessments 

Represent a broad range of assignments 
which students are engaged in; 

Allow for individual differences between 
students; 

Represent a collaborative approach to 
assessment; 

Have a goal of student self-assessment; 

Address improvement, effort and 
achievement; 

Link assessment and teaching to learning. 



Traditional Assessments 

Assess students across a limited range of 
assignments; 

Assess all students on the same dimensions; 

Assessment processes are not collaborative; 

Student self-assessments are not goals; 
Focus predominately on achievement; 

Separate learning, testing and teaching. 



In addressing the question of validity, researchers, Supovitz and Brennan, (1997) investigated whether 
alternative assessments result in greater equity than standardized tests with regard to gender, socioeconomic status, 
and race/ethnicity in the Rochester, New York school system. The authors related that their main dilemma was that 
in order to Judge which assessment is actually closer to real student performance, they needed to know each child’s 
true ability. They admitted that since they had no way of knowing the true ability of each student they cannot fully 
determine which assessment is more biased; that is, which assessment deviates farther from each child’s true ability 
in ways that are associated with the child’s gender, race/ethnicity, socioeconomic status and English language 
learner’s status. Due to this limitation, the authors could only compare the two assessments relative to each other, 
but they could not say how close either of the assessments is to an ideal situation where systematic bias is 
eliminated. Although the authors could not determine which is more biased, they found that portfolios are more 
consistent in regards to race/ethnicity, but less consistent in regards to gender. 
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Neil and Medina (1989) have also researched the question of assessment validity and have come to the 
conclusion that performance and other authentic assessments have the potential either to be fairer or to magnify 
existing inequities. Test validity is then tied to the purposes for which an assessment is used. 

According to Sacks (1999) standardized testing— especially in the high stakes arena— tend to narrow the 
curriculum by encouraging a “teach to the test” approach in the classroom. Not only is narrowing the focus of the 
curriculum a concern, but also the pressure to provide inappropriate modifications to raise scores. 

Alternative assessments have been promoted to provide a more equitable appraisal of students’ 
achievement levels. However, it is a mistake to assume that shifting from traditional standardized tests to authentic 
assessments will necessarily lead to equality of performance (Linn, Baker and Dunbar, 1991). The task then is to 
find assessment strategies that measure criterion reflecting the life experiences and values of the learner. The 
marriage of authentic assessment and authentic teaching takes on increased significance. Linn et al. mention that 
gaps in performance exist among groups because of differences in the familiarity and exposure of the students being 
assessed. The development of assessments that take into account the diversity of today’s student populations can 
make a substantial difference with many at-risk students who come to school deficient in prior knowledge that is 
important to school achievement. Linn maintains that an important outcome of the attention given to alternative 
assessment is that the educational community is reconsidering what are valid interpretations of any kind of 
assessment information. 

Educational Implications of Assessment’s Impact on Instruction 

In analyzing how assessment policies alter classroom teaching, Wong and Anagnostopoulos (1998) 
maintain that in the case of the Chicago low performing schools the district’s use of standardized reading test scores 
as the measure of school performance resulted in schools and teachers targeting resources on activities specifically 
linked to raising test scores thus affecting what teachers taught. These actions implied a connection between 
assessment strategies and pedagogy. 

Avery (1999) maintains that despite the increasing use of authentic assessments very little empirical 
research has explored the nature of authentic assessments, performance-based assessments, or their relationship to 
instruction and learning. Stecher (1998) supports the position that questions about the impact of portfolio 
assessments on school practices have not been given as much attention as other authentic assessments. Little 
evidence exists to show the effects of portfolio assessment on instructional practice. The initial research on 
portfolios has almost always focused on the quality of the information they provide, their reliability and validity and 
not their impact on teaching practices (Stecher, 1998). 

In Nystrand’s (1997) view, monologic assessment practices contribute to a setting that does not assign 
significant and serious epistemic roles to students that the students themselves can value. According to Dietel, 
Herman, and Knuth (1991) alternative assessment strategies have led the quest for more meaningful assessments to 
better capture the significant outcomes we want students to achieve and better match the kinds of tasks which they 
will need to accomplish in order to assure their future success. 

Understanding assessment terms, principles, and options is important for educators. Teachers are 
increasingly expected to develop new assessment methods and to select assessment materials skillfully. Standards 
for Teacher Competence in Educational Assessment of Students jointly developed by the American Federation of 
Teachers, the National Council on Measurement in Education, and the National Education Association in 1990, 
stated that teachers should be skilled in 

■ choosing appropriate assessment methods; 

■ developing appropriate assessment methods; 

■ administering, scoring, and interpreting assessment results; 

■ using assessment results in making decisions about students, instruction, curricula, and schools; 

■ valid grading 

■ communicating assessment results to students, parents, other educators, and other audiences; and 

■ recognizing unethical, illegal, and inappropriate assessment methods. 

Despite these increased expectations, teachers are rarely offered useful training in assessment. This 
resulted in confused attempts to improve classroom assessment. 

Factors Influencing the Implementation of a 
Comprehensive Portfolio Assessment Plan 

The factors that influenced the implementation of a comprehensive portfolio assessment plan include: 

Time and Scheduling, Roles and Responsibilities, and Technology Support. 
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Time and Scheduling 

While many reasons are given for why portfolio assessment is not practicable the most common complaints 
involve the perceived lack of time. The key to successful assessment reform, however, is not finding more time (to 
do the same things we now do) but to deploy time, so that as we are mindful of a new goal we are working 
smarter — not harder (Wiggins, 1998). Central to rethinking and deployment of time is coming to the understanding 
that non-contact time for teachers (that is, time when teachers are not obligated to students) is essential and must be 
planned for. For this to be feasible on a practical level it must be accomplished through incremental changes. 

Roles and Responsibilities 

Developing a portfolio is a team effort. This process needs to be supported by the administration, teachers, 
parents and students. As Cibulka and Derlin (1998) advocate, for an assessment system to be viewed as authentic 
“the results it produces, or seems to produce, are credible in the eyes of a variety of audiences (e.g., policy makers, 
administrators, teachers, parents, and students)” (p. 80). 

Students need to be invested in the portfolio process. Students need to have a sense of ownership of the 
process as well as the product. Portfolios should be viewed as the property of the student— not the teacher or the 
school. The teacher may use the selected pieces of the student portfolio to include in the teacher’ s own portfolio, 
but it is necessary to establish the understanding that students consider these portfolios to be their own. There are 
various ways teachers can support a feeling of student ownership. 

• Ensure students have a major say as to what is included or excluded; 

• Maintain easy physical (or electronic) accessibility of the portfolios for students; 

• Ask students’ permission to share or look over their portfolio; 

• Develop in classmates a respect for one another’s portfolios; 

• Make joint decisions on developing portfolios, sharing and other matters relating to the portfolio process 

(Tierney et ak, 1991). 

Technology Support 

Technology presents a means and an opportunity for students and professional educators to reach out to the 
global community. Portfolios are vehicles for one to utilize his/her creative talents to exhibit authentic assessment. 
The most common format for a portfolio has been a three-ring binder into which a collection of carefully selected 
artifacts was organized. Although this format has its advantages, the electronic portfolio is rapidly gaining 
popularity. 

By combining the portfolio process with technology, the uses are many and varied. An electronic portfolio 
can be viewed by a single user on a personal computer as well as displayed to a large group using computer 
presentation equipment. They take up a significant amount of computer memory, especially pictures, sound, and 
video clips. To copy an electronic portfolio for backup purposes or distribution to prospective reviewers, a high 
volume storage device and/or a writeable CD must be employed. In addition to sending the storage device 
containing the portfolio to prospective reviewers, electronic portfolios can be posted to personal or school web sites 
that are accessed through the Internet. 

As with most technological applications, learning how to develop an electronic portfolio requires initial 
time and energy. However, once the portfolio is properly developed, revisions can quickly and easily be made. The 
electronic portfolio then becomes a dynamic vehicle that students and professional educators will continuously 
improve as their qualities continue to grow and change. 

Effective Portfolio Management 

The design of an effective portfolio system should be attentive to four critical areas: 

1 . Documenting, assessing, and reporting individual achievement so as to support valid inferences of 
their achievement. 

2. Documenting and reporting the quality of the school’s educational program. 

3. Providing an assessment program that supports and protects the school’s ideology and practices. 

4. Providing a coherent structure with built-in flexibility to accommodate diversity among students and 
teachers. 
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Guide for Ongoing Portfolio Development 

Figure 1 was developed as a suggestive guide for ongoing collaborative portfolio development, 
refinement, reflection, analysis, and presentation. 

Figure 1 

Guide for Ongoing Portfolio Development 
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Conclusion 

One implication of the research findings is that all of us who develop, use, or evaluate educational 
assessments need to expand our repertoire of epistemological strategies and consider alternative models for 
warranting validity conclusions— models that might support teachers’ interpretations of their students’ learning based 
on individual portfolios and recorded observations (Moss, 1992). In addition to considering the social consequences 
of assessment-based interpretations and actions, we need to consider the social consequences of the methods by 
which we warrant those interpretations and actions. 

Successfully implementing portfolios as comprehensive assessments is complex and involves a myriad of 
issues. Despite the potential that portfolio assessments offer, the intellectual and resource demands of developing a 
portfolio assessment strategy are immense. Managing and planning the portfolio process is an ongoing activity. One 
must also realize that “by embracing alternative assessments, educators are beginning rather than ending a complex 
process’’ (Baker, 1994, p.58). As Kohn (2000) advocates, the question of appropriate assessments represents a series 
of decisions that should be questioned and revisited. 

References 

Avery, P.G. (1999). Authentic assessment and instruction. Social Education, 63,368-373. 

Baker, E. (1994). Making performance assessment work: The Road Ahead. Educational Leadership, 5\, 
58 

Baker, E., Ereeman, M., & Calyton, S. (1990). Cognitive assessment of subject matter: Understanding the 
marriage of psychological theory and educational policy in achievement testing. Los Angeles, CA: UCLS Center 
for the Study of Evaluation. Tech. Rep. No. 308. 

Cibulka, J., & Derlin, R. (1998). Authentic education accountability policies. In R. J. S. MacPherson (Ed.), 
The politics of accountability: Educative and international perspectives, (pp.79-92). Thousand Oaks, CA: Corwin 
Press. 

Dietel, R. J., Herman, J. L., & Knuth, R. A. (1991). What does research say about assessment? Oak 
Brook, IL: NCREL. 

Henke, R., Chen, X., and Goldman, G. (1999) What happens in Classrooms? Instructional practices in 
Elementary and Secondary Schools, 1994-95, National Center for Educational Statistics. NCES 1999-348. Project 
Officers, Rollefson, M. and Gruber, K. U.S. Department of Education. Washington, DC: Government Printing 
Office. 

Kohn, A. (2000). Burnt at the high stakes. Journal of Teacher Education, 51,315-327. 

Linn, R.L., Baker, E.L., & Dunbar, S.B. (1991). Complex, performance-based assessment: Expectations 
and validation criteria. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student 
Testing (CRESST). SCE Report 331. 

Moss, P. (1992). Shifting Conceptions of Validity in Educational Measurement: Implications for 
Performance Assessment. Review of Educational Research, 62 (3), 229-258. 

Neil, M., & Medina, N. (1989). Standardized testing: Harmful to educational health. Phi Delta Kappan, 

70, 688-697. 

Nystrand, M. (1997). Opening dialogue: Understanding the dynamics of language and learning in the 
English classroom. New York: Teachers College Press. 

Sacks, P. (1999). Standardized Minds. Cambridge, MA: Perseus Publishing. 

Stecher, B. (1998). The local benefits and burdens of large-scale portfolio assessment. Assessment in 
Education, 5, 335-351. 

Supovitz, J., & Brennan, R. (1997). Mirror, mirror on the wall, which is the fairest of all? An examination 
of the equitability of portfolio assessment relative to standardized tests. Harvard Educational Review, 67, 472-506. 

Tierney, R. J., Carter, M. A., & Desai, L. E. (1991). Portfolio Assessment in the Reading-Writing 
Classroom. Norwood, MA: Christopher-Gordon Publishers, Inc. 

Wiggins, G. (1998). Educative Assessment: Designing Assessments to Inform and Improve Student 
Performance. San Erancisco: Jossey-Bass. 

Wong, K., & Anagnostopoulos. (1998). Can Integrated Governance Reconstruct Teaching? In R. J. S. 
MacPherson (ed.). The Politics of Accountability: Educative and International Perspectives, (pp. 26-42). Thousand 
Oaks, CA: Corwin Press. 



484 




